Creating lambda layers using AWS IaC tools | DoingCloudStuff

Creating lambda layers using AWS Infrastructure as Code (IaC) tools

author: Vincent Chan

In this post, I describe how to create lambda layers using the different AWS Infrastructure as Code tools, as well as their pros and cons and potentially other random thoughts thrown into the mix.

What is Infrastructure as Code?

Really, Infrastructure as Code (or IaC for short) is pretty much what you would expect given the name. It is the practice of having your infrastructure (that is, whatever AWS resources you have deployed) be represented using code. And, since the infrastructure is already represented by your code, you can (or should be able to) deploy your infrastructure just by executing your code.

There are many different IaC tools out there, but the ones I personally hear about the most are

  1. Terraform,
  2. AWS Cloudformation,
  3. Serverless,
  4. AWS SAM (Serverless Application Model), and
  5. AWS CDK (Cloud Development Kit).

Of those 5, only Terraform is not locked to a particulary cloud platform (the rest are specific to AWS). For this reason, Terraform can be particularly attractive to those who wish to avoid vendor lock-in. As I am not concerned (not at the moment anyway), I haven't had reason to pick up Terraform on top of the other tools I know.

Also, Serverless includes monitoring and CI/CD capabilities as well as plain-old IaC capabilities, hence why it describes itself as a Framework instead.

Alas, as I am yet unfamiliar with Terraform and Serverless, I will be focusing on the other three tools instead:

  • AWS Cloudformation,
  • AWS SAM, and
  • AWS CDK.

Lastly, AWS CDK is unqiue among these 5 in that, instead of coding in JSON or YAML, you use actual programming languages (currently, you get your choice of Typescript, Python, .NET, and Java).

Creating lambda layers using AWS Cloudformation

The Cloudformation resource type for a lambda layer is AWS::Lambda::LayerVersion (see their docs here for more information).

As described by the Cloudformation docs for AWS::Lambda::LayerVersion | The AWS::Lambda::LayerVersion resource creates a Lambda layer from a ZIP archive. we can tell that Cloudformation expects us to have created a ZIP archive containing the code for the lambda layer (btw, in the correct structure as well) beforehand.

The syntax according to the AWS docs is

Cloudformation lambda layer syntax
Type: AWS::Lambda::LayerVersion
Properties:
  CompatibleRuntimes:
    - String
  Content: Content
  Description: String
  LayerName: String
  LicenseInfo: String

Further, the AWS::Lambda::LayerVersion Content syntax is

AWS::Lambda::LayerVersion Content syntax
S3Bucket: String
S3Key: String
S3ObjectVersion: String

And, so, we can further tell that the ZIP archive should already exist in an S3 bucket.

So, supposing you have your lambda layer code in S3 bucket my-s3-bucket and located at /lib/my-lambda-layer.zip, your Cloudformation resource would look like

Cloudformation lambda layer example
MyLambdaLayer:
  Type: AWS::Lambda::LayerVersion
  Properties:
    CompatibleRuntimes:
      - Python3.7
      - Python3.8
      - Python3.9
    Content:
      S3Bucket: my-s3-bucket
      S3Key: lib/my-lambda-layer.zip
    Description: My lambda layer
    LayerName: my-lambda-layer

Creating lambda layers using AWS SAM

The AWS SAM version (see docs here) looks pretty similar to the Cloudformation version (which is expected since AWS SAM is an extention of sorts of AWS Cloudformation). However, in our case, the key difference is that, instead of having you create the zip file and uploading it to an S3 bucket yourself, SAM handles that for you.

The syntax for AWS SAM's AWS::Serverless::LayerVersion is

AWS::Serverless::LayerVersion syntax
Type: AWS::Serverless::LayerVersion
Properties:
  CompatibleRuntimes: List
  ContentUri: String | LayerContent
  Description: String
  LayerName: String
  LicenseInfo: String
  RetentionPolicy: String

Supposing you have your Python module code located (relative to where you're executing your sam deploy command) at lib/my-lambda-layer/python, then your SAM resource (initial version) might look like

AWS SAM lambda layer example (version 1)
MyLambdaLayer:
  Type: AWS::Serverless::LayerVersion
  Properties:
    CompatibleRuntimes:
      - Python3.7
      - Python3.8
      - Python3.9
    ContentUri: ./lib/my-lambda-layer
    Description: my-lambda-layer

Notice how even though the Python module code is contained in lib/my-lambda-layer/python, the ContentUri in the SAM template is ./lib/my-lambda-layer. This is because lambda layers expect a certain file structure. The exact structure depends on the runtime (Python vs node vs java vs ...), but, for Python, it expects all of your modules be contained in a top-level python/ directory. (Actually, this isn't the whole truth as there is an alternative structure you can adopt for Python lambda layers, but I find that one cumbersome and so won't discuss it any further.)

Annoyances and fixes

Hopefully, you've noticed that I've added (version 1) to the SAM lambda layer example code snippet. That is because I find the above solution lacking. As is, you would need to either

  • manually install the modules into lib/my-lambda-layer/python yourself every time before deploying or
  • install it once and keep the files as part of your git repository.

I'm not a fan of either.

Luckily, there is a third option and one that's described in the AWS docs (see here): make use of the Metadata resource attribute.

That is, upgrade the above example to

AWS SAM lambda layer example (final version)
MyLambdaLayer:
  Type: AWS::Serverless::LayerVersion
  Properties:
    CompatibleRuntimes:
      - Python3.7
      - Python3.8
      - Python3.9
    ContentUri: ./lib/my-lambda-layer
    Description: my-lambda-layer
  Metadata:
    BuildMethod: makefile

and SAM will know to use the Makefile you provided (Makefile should be found in lib/ in this example) to "build the layer." More specifically, SAM will expect that your Makefile contains the needed command (build-MyLambdaLayer in this example) to install the modules (the ones you want as part of your lambda layer) to the correct location (in our example, lib/my-lambda-layer/python/).

Makefile
build-MyLambdaLayer:
    mkdir -p "$(ARTIFACTS_DIR)/python"
    cp *.py "$(ARTIFACTS_DIR)/python"
    python -m pip install -r requirements.txt -t "$(ARTIFACTS_DIR)/python"

As a last note, if you follow this approach, I would also recommend that you add the following to your .gitignore file

lib/*/python/*

Creating lambda layers using AWS CDK

I find that much of the syntax for AWS CDK is actually very similar to that of AWS SAM. The main benefits I can tell for AWS CDK is that

  • you get to program it in a scripting language rather than JSON or YAML and
  • permissions are much easier to deal with (at least while following the Least Privilege Principle).

Because you have a few choices of programming languages to program AWS CDK in, AWS provides separate docs for each language. That said, their syntaxes are all nearly identical (actually, I've only used Typescript and Python, so I'm just making assumptions about the rest) and, so, I've found myself checking out the documentation for Typescript instead of for Python most of the time, despite programming it in Python.

Btw, the AWS CDK API Reference doc can be accessed here.

Without further ado, let's check out how AWS CDK deals with lambda layers.

Resources are represented as classes in AWS CDK and you can read the syntax for creating lambda layers for Typescript here and for Python here.

Assuming Python and that you've imported the necessary statements (I'm assuming you have experience with AWS CDK here), a lambda layer would be represented as

AWS CDK lambda layer example
# from aws_cdk import (
#   aws_lambda as lambda_,
#   core as cdk,
# )

# the following is defined inside a cdk.Stack class' `__init__` method
my_lambda_layer = lambda_.LayerVersion(
    self,
    "MyLambdaLayer",
    code=lambda_.Code.from_asset("./lib/my-lambda-layer"),
    compatible_runtimes=[lambda_.Runtime.PYTHON_3_7, lambda_.Runtime.PYTHON_3_8, lambda_.Runtime.PYTHON_3_9],
    layer_version_name="my-lambda-layer",
)

Note: One thing that's nice about CDK is that the CDK modules contain type hinting, so, if you're using an IDE that includes intellisense and autocomplete, it can really help you developer your IaC.

However, much like for AWS SAM, AWS CDK assumes that the code for the 3rd party modules are contained in the exepected location from the start and I don't like that. Further, unlike AWS SAM, AWS CDK doesn't contain a Metadata parameter for lambda_.LayerVersion, so we need to find some other solution.

What I eventually came up with was the following.

Additional context

First, the command for deploying a CDK script is

npx cdk deploy

This command will look for a cdk.json file and, within that file, the app field which will in turn contain the command for executing the Python AWS CDK script.

cdk.json
{
  "app": "python app.py"
}

So, here, you can see that the command it triggers is python app.py. app.py looks something like

import os
from aws_cdk import core as cdk
from cdk.main_stack import MyStack

app = cdk.App()

MyStack(app, "mystack")

app.synth()

and it is within the class MyStack's __init__ method where the above AWS CDK lambda layer example lies.

Lastly, the files cdk.json and app.py are both contained in the same level, directory-wise.

My awkward solution

My solution is to modify the cdk.json file to

cdk.json
{
  "app": "make && python app.py"
}

which will tell AWS CDK to first execute the Makefile contained in the same directory cdk.json is in and then run python app.py. As such, we just need our Makefile install the desired third-party modules into the directory that AWS CDK expects to them to be in. For our example, suppose that we have a requirements.txt file located in ./lib/my-lambda-layer, then the Makefile would look something like

all:
    cd lib/my-lambda-layer
    python -m pip install -r requirements.txt -t python/

clean:
    rm -rf lib/my-lambda-layer/python

As a last note, if you follow this approach, I would also recommend that you add the following to your .gitignore file

lib/*/python/*