Creating Lambda Layers for AWS Lambda Functions with Terraform

This article assumes you have some knowledge on how to write and deploy Terraform code with AWS. If you don't have Terraform with AWS already up and running, please have a look at Terraform with AWS. You should also have some experience with AWS Lambda functions. This article does not go deep into how a Lambda function works, but I will be explaining how to set up a Lambda function. For more information about Lambda functions, see AWS Lambda documentation and Lambda Terraform documentation.

Our use case

Let's say you want to use a Lambda function, running Python to do some operations. By default, Lambda functions only support some basic Python packages during runtime. In our use case, we want to import additional packages to use in our code. The solution to our problem is called Lambda Layers. For our example, I have chosen to install a package called requests. This is a common package that allows you to send HTTP requests using Python. Note, this is just an example. You could install any Python package you want.

Lambda Layers

A Lambda Layer is a zip file archive that stores our dependencies and packages so that they can be used during runtime. These layers can be shared between multiple Lambda functions. A Lambda layer has all packages and dependencies bundled up into a single zip file. In this article I'll demonstrate this using a python package called requests. Note, lambda layers is not exclusively for Python, but also works with other programming languages supported by AWS Lambda.

First, let's look at our Python function file.

import requests
    
def lambda_handler(event, context):
    x = requests.get('https://spak.no/no/costmanagement')

    print(x.text)

As you can see, here I use the Python package requests which is not included in AWS Lambda by default. Obviously, this is only a code snippet to demonstrate, without any real-world application.

I need to zip this file in order for Terraform to use this as our Lambda function. I'll zip this and call it my_lambda_function.zip.

Prepare our Lambda Layer

Next step is to create our zipped file for our Lambda Layer to use. I'll be using a Python package manager called pipenv to install the package, but you could use others like e.g pip. Here's a link to Pipenv documentation.

First, I'll open up a folder on my computer and create a Pipenv environment, using the pipenv shell command.

Here, I'll also create the following directory structure

python/lib/python3.8/site-packages/

Now, I run pipenv install requests to install the required packages needed. Now, I want to take these packages and put them in the directory I just created. On MacOs, these packages are stored under:

~/Users/username/.local/share/virtualenvs/our-pipenv-vm/lib/python3.8/sitepackages

For other OS, the Pipenv documentation should explain where these files can be found.

Now let's copy the following files to the Python directory I created earlier: requests, charset_normalizer, idna and certifi. Remember to also copy over the corresponding *-dist-info directories to the all the 4 packages I copied over. You should have a total of 8 directories under python/lib/python3.8/site-packages/ that was created earlier. According to the AWS documentation, you can choose between storing your package files under

/python

/python/lib/python3.8/site-packages

Note: In my experience, Lambda does not always pick up on packages stored directly under

/python

so I recommend using the whole path, like below.

.
+--python
   +--lib
      +--python3.8
         +--site-packages
            +--requests/
            +--requests-2.28.2.dist-info/
            +--charset_normalizer/
            +--charset_normalizer-3.0.1.dist-info/
            +--idna/
            +--idna-3.4.dist-info/
            +--certifi/
            +--certifi-2022.12.7.dist-info/

Now, this could be done with any Python packages I might want to install. The final step I have to perform now is to zip this whole directory and give it a name e.g python_requests.zip. On MacOs the command is:

zip -r python_requests.zip python/

Creating Lambda Layers in Terraform

Now that I've prepared my Python function and Lambda Layer file, I can start writing Terraform code. Before I explain the Terraform, I'd like to show what our code-base tree structure should look like:

.
+--iam.tf
+--lambda.tf
+--lambda_layers.tf
+--my_lambda_function.zip (file I zipped earlier.)
+--layers
   +-- python_requests.zip (Python directory that I zipped earlier.)

(Note: I've specifically chosen to exclude other necessary Terraform-related files to avoid confusion.)

First, I start of by creating my Lambda function. Create a file, lambda.tf, and insert the following code.

# My Lambda Function
resource "aws_lambda_function" "my_lambda_function" {
    filename      = "my_lambda_function.zip"
    function_name = "MyLambdaFunction"
    role          = aws_iam_role.lambda_role.arn
    handler       = "my_lambda_function.lambda_handler"
    source_code_hash = filebase64sha256("my_lambda_function.zip")
    runtime = "python3.8"
    layers = [aws_lambda_layer_version.python38-requests-layer.arn]
}

Here I set the following parameters for our Lambda function.

filename: The filename of the zip file I want to use as our Lambda function.
function_name: A unique name for our Lambda function.
role: The IAM role our Lambda function will use. Every Lambda function is required to have a IAM role. I'll be creating this role in the next step.
handler: The name of our method in our code that processes events and is triggered when our Lambda function is invoked. Here I use my_lambda_function (name of file containing our code.) and lambda_handler (name of our main method).
source_code_hash: The path to our local source archive file that contains our functions code.
runtime: The identifier of our functions runtime. Here I use python3.8.
layers: A list of layers I want to attach to our Lambda function. I'll create these later in the article.

Now, lets create our IAM role. Create a file called iam.tf and add the following code.

resource "aws_iam_role" "lambda_role" {
  name = "my-lambda-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

Here I can attach permissions to my Lambda functions if needed. See AWS documentation for more information. Since Lambda Layers don't require any permissions, I'll leave this empty for this article.

The next step is to create our Lambda Layers. Create a file, lambda_layers.tf, and add the following.

# Python requests layer
resource "aws_lambda_layer_version" "python38-requests-layer" {
    filename            = "layers/python_requests.zip"
    layer_name          = "python_requests"
    source_code_hash    = "${filebase64sha256("layers/python_requests.zip")}"
    compatible_runtimes = ["python3.8"]
}

Set the following parameters.

filename: The path to the layers deployment package. I've chosen to put all layers under layers/ in my code-base.
layer_name: A unique name for our layer.
source_code_hash: The path to our Python package. This is mainly used to trigger changes.
compatible_runtimes: A list of compatible runtimes. I only want python3.8 here.

Now, I can run terraform apply and see our Lambda function with our Layer attached to it and my Lambda function should have access to the requests package I created.

Some considerations when creating layers. A lot of times, I want to re-use layers for many Lambda functions. Therefore it might be a good idea to make each layer specific to one or two python packages. This way we can cherry-pick the packages we need for a specific function. If I were to put all packages in one single layer, chances are, that some of these packages might be redundant for certain Lambda functions. The bigger layer you have, the slower your function will cold-start.

Summary

In this article, I've showed how to create a custom Lambda Layer and attach it to my Lambda function created in Terraform. I've also explained briefly how Lambda Layers work and given some best practice advice when creating and using Lambda Layers in a functioning environment with multiple Lambda Functions.