AWS: The State of Account State

 In September 2025, AWS announced that the Account information in the Organizations Service will have a new State field to replace the Status field. Since that date, both fields are available for all Organizations operations, but the Status field is vowed to be removed on September 2026.

When you read such an announcement and you know your code is using the Status field, you project to review your code and update it. So we did quite immediately, but we could not see the new State field when executing our lambdas. So we postponed the update for later.

Recently, I had another look at the problem, and still could not see any State field appearing in lambdas. I tested some call to DescribeAccount within CloudShell, but the field was really there. So I decided to run the following lambda:

import boto3

import botocore

def lambda_handler(event, context):

    print("boto3:", boto3.__version__)

    print("botocore:", botocore.__version__)

    org_client = boto3.client("organizations")

    response = org_client.describe_account(AccountId="123456789012")

    print(response)

I was surprised by the result.

boto3: 1.40.4
botocore: 1.40.4

Those versions were released in August 2025, before the update. CloudShell in my test uses botocore 1.42.72, which is from March this year. When I notified AWS Support about it, they just told me to use a Layer with a more recent botocore included. How long should I keep this temporary workaround?

AWS: Duplicates in Search Provisioned Products

 Using our beloved boto3 library, we are looking for the list of all our Provisioned Products in Service Catalog.

sc_client = boto3.client('servicecatalog')

result = sc_client.search_provisioned_products(

    PageSize=20

)

I won't bore you with th code that loops over the result and perform the operation again if we have more than 20 products. But the strange thing, is that wherever we had more than the page size, some products were repeated in the other pages. Thinking of a bug in AWS Service Catalog, we reached out to the Support Team. This is their answer:

This is a known behavior with the SearchProvisionedProducts API when using the default relevance-based sorting. Because results are sorted by relevance, the ordering can shift slightly between paginated requests, which causes duplicates (or occasionally missed items) across pages.

Never heard of relevance-based sorting. Looking at the documentation, there is no mention of it:

SortBy

The sort field. If no value is specified, the results are not sorted. The valid values are arn, id, name, and lastRecordId.

 Then, Support Team is proposing a solution:

Adding SortBy='createdTime' gives the pagination a stable ordering, so the page token points to a consistent boundary between pages. No more duplicates should appear regardless of how many provisioned products you have.

It is interesting to note that 'createdTime' is not listed in the documentation either. We tried it and it works. So a hidden feature solves a known bug.

AWS: Find Root Cause of Failure for CloudFormation Stacks

 When a CloudFormation stack fails, you have to scroll back trough the events to find the root cause of the failure. Recently, AWS even added a "Detect Root Cause" button to the Console to immediately scroll to the correct event. But how do you do it from a python script?

import boto3

def find_root_cause(stack_name):

    cf_client = boto3.client('cloudformation')

    next_values = "First Time"

    params = {

        "StackName": stack_name

    }

    root_cause = None

    while next_values:

        result = cf_client.describe_stack_events(**params)

        next_values = result.get("NextToken")

        params["NextToken"] = next_values

        for event in result["StackEvents"]:

            status = event.get("ResourceStatus", "")

            reason = event.get("ResourceStatusReason")

            # start of deployment

            if reason == "User Initiated":

                return root_cause

           

            if reason and "FAILED" in status:

                root_cause = reason

    return root_cause

You follow the same pattern as from the Console. You go back the events history, until you reach the oldest error message before the start of the deployment.

Moto: Alias Issue when Creating S3 Access Point is Fixed

 Two days ago, I discovered a small bug in the moto library that I use to unit test my lambdas on AWS. I needed to create an S3 Access Point with boto3, and while retrieving the its alias, I had different results when using the return value of create_access_point or get_access_point.

So I wrote a small unit test:

from moto import mock_s3control

import boto3

@mock_s3control

def test_access_point_alias():

    client = boto3.client("s3control")

    alias_from_create = client.create_access_point(

        AccountId="123456789012",

        Name="my-access-point",

        Bucket="MyBucket",

    )["Alias"]

    alias_from_get = client.get_access_point(

        AccountId="123456789012",

        Name="my-access-point",

    )["Alias"]

    assert alias_from_create == alias_from_get

I create an S3 Access Point, and retrieve its alias in two ways: from the response of the create_access_point function, and from the get_access_point function. On my moto 4.2.6, this test fails.

So I opened an issue on the project's repository. It was fixed and closed on the same day. That's reactivity!

AWS: The Next Token Pattern

When developing  for AWS, there is a pattern that you use each time a response to a service may return a lot of data. You get back some of the data, together with a token that you can provide to get another round of data.

As an example, let's take the service that returns the list of events from a Cloudformation template deployment. Here is how you would do it using Python and boto3:

cf_client = boto3.client("cloudformation")

response = cf_client.describe_stack_events(StackName="mystack")

# do something with the response

while response.get("NextToken"):

    response = cf_client.describe_stack_events(

        StackName="mystack",

        NextToken=response.get("NextToken")

    )

    # do again something with the response

However, there is one thing that I do not like with this pattern: code repeat. You call the service at two different parts of your code, with almost identical parameters. And you process the response in the same way, again in two places. If you have to fix something in this code, you have to remember to fix in both places.

The approach I use to have your code only once, is to take advantage of Python's capacity to pass parameters as a dictionary. Here is my approach to this pattern:

cf_client = boto3.client("cloudformation")

next_token = "FIRST TIME"

params = {"StackName": "mystack"}

while next_token:

    response = cf_client.describe_stack_events(**params)

    # do something with the response

    next_token = response.get("NextToken")

    params["NextToken"] = next_token

I store my parameter list in a dictionary, and I initialize the next token with something that is not empty. So the first time in the loop will always run. I can then call my service, without the token. After processing the response, I then read the next token and fill it in my parameter list. The second time round, it will call the service with my token.

Something funny happened when I started using this pattern into our production code. We tried using Bandit, which is a tool that analyze your code and looks for security issues. It would systematically flag my pattern with this error: 

[B105:hardcoded_password_string] Possible hardcoded password: 'FIRST TIME'

 Well, I have to slightly modify my pattern to avoid using the word token...

AWS: Simpler S3 File Deletes by Prefix

 I came across this code that deletes files in an S3 from a list of prefixes:

s3_client = boto3.client('s3')

for prefix in prefix_list:

    paginator = s3_client.get_paginator('list_objects_v2')

    file_list = paginator.paginate(

        Bucket=data_bucket,

        Prefix=prefix

    )

    for current_content in file_list:

        for current_file in current_content.get('Contents', []):

            current_key = current_file['Key']

            response = s3_client.delete_object(

                Bucket=data_bucket,

                Key=current_key

            )

The code creates an S3 client, and then, for each prefix in a list, it creates a paginator. Paginators are great because they help you avoid using all your memory when the list of files is big. Using this paginator, the code retrieves the list of all the files corresponding to the prefix, and deletes it.

Nothing bad in the code, it works nicely. My only remark here, is that there exists a simpler way. Instead of using the S3 client, you can create an S3 bucket resource. From there, you can simply delete all files listed under a prefix using a simple filter:

s3_resource = boto3.resource('s3')

bucket = s3_resource.Bucket(data_bucket)

for prefix in prefix_list:

    bucket.objects.filter(Prefix=prefix).delete()

Simpler!

Warnings on SQS endpoint URL while mocking with moto

 We have many unit tests using moto library to mock calls to boto3 to access AWS services.They all work fine, except for a warning on the use of this fixture:

@fixture(scope='function')

def sqs():

    with mock_sqs():

        yield boto3.client('sqs', region_name=AWS_REGION)

This is one of the correct ways of using moto mocks. But only this mock on SQS shows the following warning:

FutureWarning: The sqs client is currently using a deprecated endpoint: eu-central-1.queue.amazonaws.com. In the next minor version this will be moved to sqs.eu-central-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.

 Now, our Sonar is considering these warnings as errors, and leads our build to fail. As explained by the issue, the warning comes from the use of a deprecated attribute called sslCommonName. I didn't dive into the moto library, but there might be a way to avoid the use of this attribute. The easisest way to stop using it, is to add this environment variable to our test code:

import os

os.environ['BOTO_DISABLE_COMMONNAME'] = 'true'

This skips completely the use of the deprecated attribute.

AWS: Unblock CloudFormation stacks from UPDATE_ROLLBACK_FAILED state

 In our project, we have lots of AWS accounts. And we are in charge of deploying base resources in each of them. To do this, we use CloudFormation to deploy several stacks. Actually several stacks nested into one master stack.

One problem we have is that our customers do not always update their stacks to the latest version. Also, when a stack fails to update, they sometimes let it rollback, and do not care to ask for a fix. Of course, this summer, there was even less updates, people being on vacation. And also no release, since we felt there would be nobody to deploy it. So we end up this September with a larger release than usual.

The result of all this is that we found ourselves staring at lots of accounts with a stack entering UPDATE_ROLLBACK_FAILED state. The reason? Deprecation. As it happened, AWS decided to deprecate Python 3.6, and also a couple of Policies (AWSConfigRole, AWSCloudTrailReadOnlyAccess). Of course we updated our stacks with the correct values for Python and the replacement policies some time ago. But as the stacks were not always up to date, and there were some issues while updating to the new release, many stacks started to rollback. And since some of the rollbacked values were deprecated, the rollbacks failed.

When you are in that case, you have two choices. The first one is to delete everything and redeploy. We tried it on one account, and it was really painful. Too many dependencies and manual actions. The second one is the one advised to us by the AWS support itself: continuing rolling back. When you continue a rollback, you have the possibilities to skip some resources. In our case, we needed to skip all lambdas using Python 3.6, and all roles using the deprecated policies.

We tried it manually in the AWS Console, and there is one caveat: you can select resources from nested stacks, but not resources from stacks nested into nested stacks. Since we have many accounts to update, and many resources to rollback, we decided to script the whole process.

We thought it will be simple: using Python and boto3, we list all the resources in our stack, recursively entering nested stacks, and filtering all lambdas and roles. We ran into several problems:

• You cannot skip resources that are not in a failed stack
• You cannot skip resources that are not in a failed state
• You cannot skip resources that are in a failed state because CloudFormation cancelled the update
• Once you run rollback with skipped resources, CloudFormation discovers new failing resources, so you have to iterate until all is fine, or the list of resources to skip does not change between two iterations.
• Name of resources in nested stack are <nested_stack_name>.<resource_logical_id>. Even for resources in several level of nested stack, you still use the same pattern, giving only the name of the direct parent stack.
• Waiting for a rollback to complete will throw an exception if the rollback fails.

This is the script that helps turning a stack from UPDATE_ROLLBACK_FAILED to UPDATE_ROLLBACK_COMPLETE:

import boto3

BLOCKABLE_RESOURCES = [

    "AWS::Lambda::Function",

    "AWS::IAM::Role",

]

STACK_NAME = "MyStack"

def get_stack_status(cf_client, stack_name):

    response = cf_client.describe_stacks(StackName=stack_name)

    return response["Stacks"][0]["StackStatus"]

def find_blocking_resources(cf_client, stack_name, parent, resources):

    response = cf_client.describe_stack_resources(StackName=stack_name)

    if (

        parent

        and get_stack_status(cf_client, stack_name) != "UPDATE_ROLLBAK_FAILED"

    ):

        return

    for resource in response["StackResources"]:

        if (

            resource["ResourceType"] == "AWS::CloudFormation::Stack"

            and resource["ResourceStatus"] == "UPDATE_FAILED"

        ):

            nested_name = resource["PhysicalResourceId"].split("/")[1]

            find_blocking_resources(

                cf_client, nested_name, nested_name + ".", resources

            )

        elif (

            resource["ResourceType"] in BLOCKABLE_RESOURCES

            and resource["ResourceStatus"] == "UPDATE_FAILED"

            and resource.get("ResourceStatusReason")

            != "Resource update cancelled"

        ):

            resource.append(parent + resource["LogicalResourceId"])

cf_client = boto3.client("cloudformation")

status = get_stack_status(cf_client, STACK_NAME)

if status != "UPDATE_ROLLBACK_FAILED":

    print("Nothing to unblock. Exiting")

    exit(0)

resources = []

previous_resources = ["DUMMY"]

waiter = cf_client.get_waiter("stack_rollback_complete")

print("Starting unblocking process")

while status != "UPDATE_ROLLBACK_FAILED" and previous_resources != resources:

    previous_resources = resources

    resources = []

    find_blocking_resources(cf_client, STACK_NAME, "", resources)

    print("Skipping ", resources)

    cf_client.continue_update_rollback(

        StackName=STACK_NAME, ResourcesToSkip=resources

    )

    try:

        waiter.wait(StackName=STACK_NAME)

    except Exception as err:

        print(err)

    status = get_stack_status(cf_client, STACK_NAME)

print("Final stack status:", status)

Of course, once this is done, you still have to fix the stacks and run an update.

Modifying AWS resource for testing

 If you are writing unit tests using boto3 and moto, you might want to modify a value in a resource. I had this case, for instance, where I wanted to set the state of an AMI to pending for a test. Using moto, AMIs are always created in the available state. However, if you try to do it this way, it will fail:

ec2 = boto3.resource("ec2")

image = ec2.Image(AMI_ID)

image.state = "pending"

You will end up with this message:

AttributeError: can't set attribute 'state'

It is possible to modify the resource object. However, it will just modify this object instance, the model itself will not be updated. So if you use the resource as a parameter to a function, it will work. But if the function retrieves its data with boto3, it will get the original value.

Here is my solution:

image.meta.data.update({"State": "pending"})

That helped in my case. 

AWS: Read Timeout when Invoking Lambda

 I have an AWS Lambda that invokes another Lambda synchronously. Nothing was wrong with it until recently, when the other Lambda started performing more tasks and taking more time. Suddenly, the caller Lambda started failing with this weird message:

ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/MyOtherLambda/invocations"

Looking at the logs of my other Lambda, I could see that it ran fine, although it was executed several times. After some research on the Internet, I found this article from AWS support that explains that when invoking a Lambda synchronously, there is a default timeout of 60 seconds, and 3 retries. This can be configured when creating the client.

In Python using boto3 for instance, you have to use the Config object:

import boto3

from botocore.config import Config

lambda_client = boto3.client("lambda", config=Config(read_timeout=600))

Now, my invoked lambda has 10 minutes to perform its tasks.

Find reasons of failed DynamoDB Transaction with boto3

 When working with Transactions in DynamoDB, you pack several actions into one query. When one of them failed, all the queries are rolled back. But how do you know which one failed? Reading the documentation, you find the following note:

If using Java, DynamoDB lists the cancellation reasons on the CancellationReasons property. This property is not set for other languages. Transaction cancellation reasons are ordered in the order of requested items, if an item has no error it will have NONE code and Null message.

 Why only in Java? I'm working in Python, using the boto3 library. Do I still have a way to access the information? As it turns out, there is a workaround. When I print the message of  my exception, I see the following:

botocore.errorfactory.TransactionCanceledException: An error occurred (TransactionCanceledException) when calling the TransactWriteItems operation: Transaction cancelled, please refer cancellation reasons for specific reasons [None, ConditionalCheckFailed, None]

 The reasons array is printed at the end of the message. If it is already there, why not have it in a more accessible format? From there, it is not difficult to extract it:

import re

def reasons(e):

    m = re.search("\\[(.*)\\]", str(e))

    reasons = m.group(1).split(", ")

    return [None if reason == "None" else reason for reason in reasons]

Now I can use it like that:

import boto3

client = boto3.client("dynamodb")

try:

    client.transact_write_items(

        [

            {

                "ConditionCheck": {... }

            },

            {

                "Put": {

                    "Item": {...},

                    "ConditionExpression": ...

                }

            },

            {

                "Put": {...}

            },

        ]

    )

except client.exceptions.TransactionCanceledException as e:

    if reasons(e)[0] is not None:

        raise MissingItemError()

    raise OperationFailedError()

I hope this array will be added to the exception's response structure as a normal field in a future version. At least, the feature request exists in boto3.

Creating AWS resources in pytest fixtures with moto

 When unit testing functions that access AWS resources, the easiest way is to use the moto library. For instance, I have to test several functions that access an SQS queue, so I proceed this way:

import boto3

from moto import mock_sqs

def create_queue():

    client = boto3.client("sqs")

    queue_url = client.create_queue(QueueName="queue")["QueueUrl"]

    sqs = boto3.resource("sqs")

    return sqs.Queue(queue_url)

@mock_sqs

def test_something()

    queue = create_queue()

    ...

Since I had this create_queue() call at the beginning of most of my test functions, I wanted to make it a fixture. So i tried this way:

import boto3

from moto import mock_sqs

from pytest import fixture

@fixture

@mock_sqs

def queue():

    client = boto3.client("sqs")

    queue_url = client.create_queue(QueueName="queue")["QueueUrl"]

    sqs = boto3.resource("sqs")

    return sqs.Queue(queue_url)

@mock_sqs

def test_something(queue)

    ...

Unfortunately, this raised an error in my test functions stating that the queue does not exist. The reason for it is that the decorator @mock_sqs on the fixture would destroy my SQS queue as soon as it would leave the queue() method.

The solution is simple: do not use the mock as a decorator, but trigger it when the fixture is initializing and destroy it when it terminates. That means using a yield within the fixture to return the queue:

import boto3

from moto import mock_sqs

from pytest import fixture

@fixture

def queue():

    mock_sqs().start()

    client = boto3.client("sqs")

    queue_url = client.create_queue(QueueName="queue")["QueueUrl"]

    sqs = boto3.resource("sqs")

    yield sqs.Queue(queue_url)

    

    mock_sqs().stop()

def test_something(queue)

    ...

Notice that we have to drop the decorator on the test function as well.

Thanks for the answers to this moto issue.

Mock boto3 services not handled by moto

We have a pattern to mock AWS services in our python lambdas. First, in our lamba code, we initialize boto3 clients with properties:

import os
import boto3

@property
def REGION_NAME():
    return os.getenv("REGION", "eu-west-3")

@property
def SQS():
    return boto3.client(service_name="sqs", region_name=REGION_NAME.fget())

def lambda_handler(event, context):
    sqs_client = SQS.fget()

As you can see, the region is also a property. The reason for it is that all clients in moto are declared on the us-east-1 region.
Our unit tests are all in a test sub-folder of our lambda. To write the tests, we usually follow this pattern:

from moto import mock_sqs
from pytest import fixture

from ..mylambda import (
    lambda_handler,
    SQS
)

@fixture(autouse=True)
def prepare_test_env(monkeypatch):
    monkeypatch.setenv("REGION", "us-east-1")   

@mock_sqs
def test_mylambda():
    SQS.fget().create_queue(QueueName='MyQueue')
    
    #do the test...

We import from the lambda in the parent folder the methods to test, as well as the properties. For this to work, we have to create an empty __init__.py file in that directory. We then patch the environment variables, including the region that needs setting to us-east-1. In our test function, we have to mock the boto3 client, using the appropriate moto annotation. Then we write our test code.

In some cases, the boto3 client has no mock in moto. That is our case for Step Functions for instance (I know it is in preparation, but not yet ready as the time of the writing). For those cases, we use the following pattern:

import boto3

@property
def SFN():
    return boto3.client("stepfunctions")

In our lambda, nothing changes. We are still using a property. However, in the unit test, we have to use a patch:

from unittest.mock import (
    PropertyMock,
    patch
)
from ..mylambda import lambda_handler

def test_lambda():
    with patch('mylambda.mylambda.SFN', new_callable=PropertyMock) as mock_stepfunctions:
        lambda_handler(event)

        mock_stepfunctions.fget().start_execution.assert_called_with(
            stateMachineArn="MyMachine", 
            name="State-Machine-0", 
            input="{}"
        )
    

We patch the property SFN with a PropertyMock. By giving it a name, we can then use the mock to assert that it was called with the correct parameters.

Send Batch of Messages to SQS

We have a simple code that can send up to several thousand messages to an SQS queue. Using Python and boto3, the code looks like this:

sqs = boto3.resource('sqs')
queue = sqs.get_queue_by_name(QueueName=SQS_QUEUE_NAME)
for message in messages:
    queue.send_message(message)

When you have really a lot of messages in an array, it is possible to send them by batch of up to 10 messages, using the send_message_batch method. When doing this, there are two problems to solve: creating the batches of 10 messages, and generating an ID for each message. AWS enforce the generation of this ID so it can send back a response containing the list of messages that failed or succeeded, identified by the ID.

Here is the new code:

sqs = boto3.resource('sqs')
queue = sqs.get_queue_by_name(QueueName=SQS_QUEUE_NAME)

for i in range(0, len(messages), 10):
    chunk = messages[i:i+10]
    queue.send_messages(Entries=[
        {
            "Id": "MSG" + str(id), 
            "MessageBody": message
        } for id, message in zip(range(10), chunk)
    ])

Our loop now jump to every tenth message. Inside the loop, we create a chunk of 10 messages, and use the send_message_batch method to send it. You can see that we use a list comprehension the runs over both the chunk and a range to generate the ID.