Running Ruby-based Lambda functions with binary dependencies can be a bit tricky because AWS Lambda expects the packages with a specific structure, and because included binaries must be built and packaged to work for the Lambda operating environment. This post describes how Bundler should be configured for Lambda and how to ensure binary dependencies are available for gems such as pg and nokogiri.

Dependency management

The AWS blog post Announcing Ruby Support for AWS Lambda blog post does an excellent job of describing how to get up and running quickly, and also includes an example on how to use RubyGems dependencies when deploying your Lambda functions. It however doesn’t address how to use RubyGems that have dependencies outside of Ruby.

The Lambda operating model requires that a Lambda function is deployed as a standalone package that contains all dependencies in a single .zip file1. For Ruby, this includes gems, binaries and other assets.

Bundler

Bundler is the primary dependency management system for Ruby2.

Your development environment is likely different to the one in production and the magic that Bundler does behind the scenes when running bundle install will be different when relying on having the correct headers, binaries and build tools for any dependencies in each environment.

For example, on my Mac I need Xcode, Homebrew and PostgreSQL installed in order to install the pg gem. On Linux these might be the postgresql-devel and "Development Tools" libraries. Bundler will also need to know where to find these dependencies. (e.g. bundle config build.pg --with-pg-config=/usr/pgsql-10/bin/pg_config).

The way to accommodate different development and production environments is to run bundle install in each of them. In a production or pre-production environment, that would mean running bundle install on the target machine after copying the code into it, e.g. onto a running machine, before generating an AMI, or inside a Docker container.

Lambda has no way to execute a command inside the environment before the code is loaded. This means we need to find a way to package and compile dependencies ahead of time in an environment that is compatible with the Lambda environment.

Emulating the Lambda environment with LambCI and Docker

docker-lambda is an emulated Lambda environment that will allow us to create and compile packages for later deployment to AWS Lambda. From their README:

A sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors

You can use it for running your functions in the same strict Lambda environment, knowing that they’ll exhibit the same behavior when deployed live. You can also use it to compile native dependencies knowing that you’re linking to the same library versions that exist on AWS Lambda and then deploy using the AWS CLI.

Example: Using pg and nokogiri gems in Lambda

Examples below are in the https://github.com/stevenringo/lambda-ruby-pg-nokogiri repository on GitHub

For convenience, shell commands described below are also in the Makefile in the repository.

Step 1: Create a LambCI-based Docker image with dependent libraries

You’ll need to create a new Docker image to install the PostgreSQL client libraries required by pg. We’ll base this on the LambCI image. In this example, I am installing the libraries for PostgreSQL 10, but this could be any version (e.g. 9.x or 11) that is compatible with the database you are connecting to.

Create a Dockerfile:

FROM lambci/lambda:build-ruby2.5

RUN yum install -y \
    https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-6-x86_64/pgdg-redhat10-10-2.noarch.rpm
RUN sed -i "s/rhel-\$releasever-\$basearch/rhel-6.9-x86_64/g" "/etc/yum.repos.d/pgdg-10-redhat.repo"
RUN yum install -y postgresql10-devel
RUN gem update bundler

CMD "/bin/bash"

Be sure to have the correct Docker tag corresponding to the required Ruby version3, and that the tag contains the build- prefix (The ones without the prefix are for testing Lambda functions, i.e. not building). I have also included an update to Bundler in the script to ensure the latest version is available.

Build your image:

docker build -t lambda-ruby2.5-postgresql10 .

You can optionally push your image to a Docker registry.

Once you have built your image you can use it with a sample

Step 2: Create a simple Ruby code sample and Gemfile

Create a new folder somewhere on your machine and create a file handler.rb:

require 'pg'
require 'nokogiri'

def main(event:, context:)
  {
    postgres_client_version: PG.library_version,
    nokogiri_version: Nokogiri::VERSION
  }
end

and Gemfile:

source "https://rubygems.org"

gem "pg"
gem "nokogiri"

Step 3: Copy dependencies and bundle install

Run the following to get a shell inside the container:

docker run --rm -it -v $PWD:/var/task -w /var/task lambda-ruby2.5-postgresql10

From inside the container, run:

bundle config --local build.pg --with-pg-config=/usr/pgsql-10/bin/pg_config
bundle config --local silence_root_warning true
bundle install --path vendor/bundle --clean
mkdir -p /var/task/lib
cp -a /usr/pgsql-10/lib/*.so.* /var/task/lib/

These commands will configure Bundler as well as install the additional libraries that the pg gem requires.

Step 4: Smoke test

Run the following inside the container:

ruby -e "require 'handler'; puts main(event: nil, context: nil)"

You should get a response that shows both gems were loaded correctly:

{:postgres_client_version=>100006, :nokogiri_version=>"1.9.1"} 

Step 5: Package to zip

You’re now ready to create a deployment package with all dependent gems and libraries for deployment to AWS Lambda.

rm -f deploy.zip
zip -q -r deploy.zip .

Step 6: Deploy

Deploy to the AWS Lambda using the console, or command line4:

aws lambda create-function \
    --region ap-southeast-2 \
    --function-name RubyLambdaPostgreSQLNokogiri \
    --zip-file fileb://deploy.zip \
    --runtime ruby2.5 \
    --role arn:aws:iam::000000000000:role/lambda-execution-role \
    --timeout 20 \
    --handler handler.main

Success!

Ruby

A follow-up post will look at how to make this process a bit easier using Lambda layers as well as in AWS Serverless Application Model and The Serverless Application Framework.


  1. Lambda also now has layers to make reuse easier, but they each require deploying their own standalone packages.
  2. Bundler will be installed by default in Ruby 2.6.
  3. Only Ruby 2.5 is available when this was written.
  4. Be sure to change the region, IAM role, etc. for your environment.