CI/CD

If you've read my first post you know I'm a big fan of Gitlab and its built in CI/CD. The hamiltix repo is set up with each lambda as a submodule because Gitlab currently does not support more than one .gitlab-ci.yml file for a repo. The gitlab-ci.yml files for each lambda are nearly identical (on purpose!), only the variables section at the top and the additional cp statements for custom directories (if needed) change between lambda functions. Strict twelve-factor followers will notice that the build and release stages are combined. It is certianlly possible to break the build step out and pass the zip as an artifact, but the stage is so fast we haven't done this yet.

variables:
  #  Set git strategy
  GIT_STRATEGY: clone
  GIT_SUBMODULE_STRATEGY: recursive
  # Keys and secrets are defined in the project CI settings and exposed as env variables
  AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
  AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
  AWS_DEFAULT_REGION: "us-east-1"
  NAME: "MyFunction"
  FILENAME: "MyFunction.py"
  HANDLER: "MyFunction.lambda_handler"
  RUNTIME: "python3.6"
  ROLE: "arn:aws:iam::XXXXXXXXXXXXX:role/XXXXXXXXXX"
  FILE: "fileb://deploy_$CI_COMMIT_REF_NAME.zip"

stages:
  - test
  - deploy

test:
  stage: test
  image: badsectorlabs/code-checking:latest # This is a docker image that contains a lot of code checking tools
  script:
    - cpd --minimum-tokens 100 --language python --files .
    # pylint output is good to look at, but not worth breaking the build over
    - pylint -d bad-continuation -d line-too-long -d import-error -d missing-docstring $FILENAME || true
    - flake8 --max-line-length 120 --ignore=E722,W503 . # You must pass flake8 (W503 is wrong, pep8 changed)

deploy-staging:
  stage: deploy
  image: badsectorlabs/aws-compress-and-deploy:latest
  variables:
    ALIAS: "Staging"
    DESC: "Staging build, commit: $CI_COMMIT_SHA"
  script:
    - virtualenv -p /usr/bin/python3.6 env
    - source env/bin/activate
    - pip install -r requirements.txt
    - mkdir dist
    - cp $FILENAME dist # copy all files needed to dist
    # Copy any other directories (modules, etc) here
    - cp -rf env/lib/python3.6/site-packages/* dist
    - cd dist
    - zip -r9 ../deploy_$CI_COMMIT_REF_NAME.zip .
    - cd ..
    - deactivate
    - ls -lart
    - echo Creating or updating $NAME
    - > # This captures the code hash for the updated/created lambda function; -r is needed with jq to strip the quotes
      CODE_SHA_256=$(aws lambda update-function-code --function-name $NAME --zip-file $FILE | jq -r ."CodeSha256" || aws lambda create-function
      --function-name $NAME --runtime $RUNTIME --role $ROLE --handler $HANDLER --zip-file $FILE | jq -r ."CodeSha256")
    - echo Publishing LATEST, CodeSha256=$CODE_SHA_256, as 'Staging'
    - VERSION=$(aws lambda publish-version --function-name $NAME --description "$DESC" --code-sha-256 $CODE_SHA_256 | jq -r ."Version")
    - echo "Published LATEST as version $VERSION"
    - >
      aws lambda update-alias --function-name $NAME --name $ALIAS --function-version $VERSION || aws lambda create-alias
      --function-name $NAME --name $ALIAS --description "Staging" --function-version $VERSION
    - echo Successfully updated $NAME:$ALIAS
  environment:
    name: master-staging
  only:
    - master

deploy-prod:
  stage: deploy
  image: badsectorlabs/aws-compress-and-deploy:latest
  variables:
    ALIAS: "Prod"
    DESC: "Prod build, commit: $CI_COMMIT_SHA"
  script:
    - virtualenv -p /usr/bin/python3.6 env
    - source env/bin/activate
    - pip install -r requirements.txt
    - mkdir dist
    - cp $FILENAME dist # copy all files needed to dist
    # Copy any other directories (modules, etc) here
    - cp -rf env/lib/python3.6/site-packages/* dist
    - cd dist
    - touch PROD # This is the canary that will tell the lambda function to use the PROD secrets
    - zip -r9 ../deploy_$CI_COMMIT_REF_NAME.zip .
    - cd ..
    - deactivate
    - ls -lart
    - echo Creating or updating $NAME
    - > # This captures the code hash for the updated/created lambda function; -r is needed with jq to strip the quotes
      CODE_SHA_256=$(aws lambda update-function-code --function-name $NAME --zip-file $FILE | jq -r ."CodeSha256" || aws lambda create-function
      --function-name $NAME --runtime $RUNTIME --role $ROLE --handler $HANDLER --zip-file $FILE | jq -r ."CodeSha256")
    - echo Publishing LATEST, CodeSha256=$CODE_SHA_256, as 'Prod'
    - VERSION=$(aws lambda publish-version --function-name $NAME --description "$DESC" --code-sha-256 $CODE_SHA_256 | jq -r ."Version")
    - echo "Published LATEST as version $VERSION"
    - >
      aws lambda update-alias --function-name $NAME --name $ALIAS --function-version $VERSION || aws lambda create-alias
      --function-name $NAME --name $ALIAS --description "Prod" --function-version $VERSION
    - echo Successfully updated $NAME:$ALIAS
  environment:
    name: master-prod
  only:
    - master
  when: manual

Using this CI setup, the lambda can check for PROD with if os.path.exists('PROD'): and if so read in env variables for the production environment, and otherwise use staging variables. Note that both staging and production variables must be defined in the lambda settings (aliases take a snapshot of the lambda settings to prevent a setting change from breaking aliases that already exist).

The CI setup for pushing the static site assets looks nearly identical to the setup for this blog.

Logging and Monitoring

Once you have some lambdas working away for you, it becomes necessary to monitor them. By default the lambdas will log any standard out to Cloudwatch, which is nice if you need to go back and see what caused an issue, but doesn't help alert you when an issue occurs. There are many ways to solve this issue, including many that would leverage AWS services but I already had a lifetime Pushover account, so decided to use it for instant push notifications on any unhandled lambda error.

def send_pushover(message, title, sound='pushover'):
    """
    Send a pushover message
    :param message: string; the message to send
    :param title: string; the title of the message
    :param sound: string; one of the keys of {'pushover': 'Pushover (default)', 'bike': 'Bike', 'bugle': 'Bugle',
                          'cashregister': 'Cash Register', 'classical': 'Classical', 'cosmic': 'Cosmic',
                          'falling': 'Falling', 'gamelan': 'Gamelan', 'incoming': 'Incoming',
                          'intermission': 'Intermission', 'magic': 'Magic', 'mechanical': 'Mechanical',
                          'pianobar': 'Piano Bar', 'siren': 'Siren', 'spacealarm': 'Space Alarm', 'tugboat': 'Tug Boat',
                          'alien': 'Alien Alarm (long)', 'climb': 'Climb (long)', 'persistent': 'Persistent (long)',
                          'echo': 'Pushover Echo (long)', 'updown': 'Up Down (long)', 'none': 'None (silent)'}
    :return: None
    """
    from pushover import init as pushover_init # install with `pip3 install python-pushover`
    from pushover import Client
    # Send Pushover notification via the API (this is the hamiltix key)
    pushover_init('XXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
    client = Client('XXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
    client.send_message(message, title=title, sound=sound)


def lambda_handler(event, context):
    try:
        return main(event, context)
    except Exception as e:
        print('[FATAL] Caught exception: {}'.format(e))
        import traceback
        error_trace = traceback.format_exc()
        print(error_trace)
        error_title = 'Error in [LambdaFunctionName]'
        send_pushover(error_trace, error_title, sound='falling')
        raise(e)  # Make sure the lambda function returns a 500

Getting a push alert any time there is an error helps us respond to issues as soon as they come up. The same send_pushover() is used to alert on other things as well, like any time a ticket is purchaced (with the cash register sound naturally).