K
Q

Running tests (pytest) in CI (gitlab-ci kubernetes runner) for AWS Lambda w/ Layer using 'sam local invoke'

12/7/2021

The reason I am trying to run sam local invoke as part of my CI pipeline is because I want to run some unit & integration tests for a lambda that uses a layer. The layer is managed in a different project and hence, its code and libraries are not directly available in my lambda function project.

With some inspiration from this post, I got the unit & integration tests working locally using sam local invoke that uses 'samtemplate_tests.yaml', which calls app_tests.py to run the unit & integration tests defined in the .tests/ folder.

#samtemplate_tests.yaml
Resources:
  MyFunctionTests:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app_tests.lambda_handler
      Layers: 
        - arn:aws:lambda:us-east-1:<account#>:layer:MyLayer:1
#app_tests.py
import pytest
def lambda_handler(event, _):
    res = pytest.main(["-x", "./tests"])
    return res

This workaround works well locally and I do not mind having to maintain two extra files (samtemplate_tests.yml, app_tests.py) in my lambda project to workaround the layer dependency.

However, getting this solution to work via CI would mean I need to enhance my CI infra to support 'docker in docker'. For CI, I am using Gitlab ci w/ kubernetes runners, which use a debian:buster build image installed w/ needed pre-reqs. I am not too familiar w/ dind and other advanced docker features, so was wondering if someone could provide direction on the approaches I could explore.

TL'DR: Is there a way to run sam local invoke as part of CI?

update: Just found this open feature request on aws-sam-cli, which would definitely help.

-- virenstack
aws-lambda-layers
aws-sam
gitlab-ci
kubernetes
pytest

Similar Questions

Spark with Kubernetes connecting to pod id, not address
Is it possible to setup multiple VirtualService that share same host in a single Gateway?
How can frontend communicate with backend if each is on a separate pod and host within kubernetes?
Approaches to multi-cloud kubernetes federation
apply HPA for Statefulset in kubernetes?

1 Answer

12/9/2021

Here's a workaround I implemented as part of my gitlab-ci 'test' job in my lambda function project to run tests via CI:

  1. Download the layer.zip
  2. Unzip to a new directory
  3. Install layer dependencies using the layer requirements.txt
  4. Copy layer python code files to the lambda function directory
  5. List item
  6. Run tests
# !/usr/bin/env bash

# script/test: Run the test suite.

set -e

cd "$(dirname "$0")/.."

if [[ $CI ]] # ci-cd build
then
    echo "==> Installing boto3 as it is required to run integration tests…"
    poetry add boto3
    
    echo "==> Downloading MyLayer zip…"
    URL=$(aws lambda get-layer-version \
            --layer-name arn:aws:lambda:us-east-1:<account#>:layer:MyLayer \
            --version-number 6 \ #TODO: get version dynamically
            --query Content.Location \
            --output text)
    curl $URL -o layer.zip
    
    echo "==> Unzipping My Layer…"
    unzip -qq -o layer.zip python/* -d Mylayer

    echo "==> Installing python dependencies required for Mylayer…"
    sed -i 's/\;.*//' Mylayer/python/requirements.txt # Gets rid of string after ";" from the poetry generated requirements file.
    poetry add `cat Mylayer/python/requirements.txt`

    echo "==> Moving Mylayer python files to lambda src directory…"
    find ./Mylayer/python/ -maxdepth 1 -type f -exec mv {} src/ \;

    echo "==> Running tests…"
    poetry run python -m pytest tests

else # local/dev build
    echo "==> Building SAM artifacts for unit & integration tests lambda…"
    sam build --config-env $1 --template-file samtemplate_tests.yaml

    echo "==> Invoking lambda func to run unit & integration tests…"
    sam local invoke MyLambdaFunctionTests \
            --env-vars=src/.envs/test_env_dev.json
fi
-- virenstack
Source: StackOverflow