How to automate build and deployment of elixir app using CI/CD

Introduction

One of the common ways to deploy an elixir-phoenix application is through releases. A release is a self-contained directory with all of your application code, all dependencies, and the whole of the Erlang Virtual Machine runtime environment. Once you package your app this way, it can be deployed to the target server environment, as long as the target server runs on the same OS distribution and version as the build machine.

In this article, I'm going to take you through how to set up a build, test and deployment pipeline for your Elixir (specifically Phoenix app) app using CircleCI and Git.

What we intend to achieve by the end of this article:

Set up the production environment
Create a CI/CD pipeline
Create a CircleCI account
Connect your GitHub or Bitbucket repo to your CircleCI project
Configure your CircleCI project
Deploy your app from your local environment to GitHub
CircleCI builds your release and deploys it to your target production server and starts the app.

So let's jump right into the actual task. We won't follow the steps one after another, but I will make sure you can track every step throughout the article.

Requirements:

You'll need to have a Phoenix application in your development environment or clone one here.

Setting up the production server

We need to create a non-sudo user for deployment. Please avoid the use of root or any user with sudo privileges.

For this step, you can access your server through ssh. I'll name the user circleci as we will use this user to perform our tasks with CircleCI.

Create a passwordless user and assign them a home directory using -m to create and -d to assign the home directory.

$ sudo useradd -m -d /home/circleci -s /bin/bash circleci

Do not add the user to sudo group as we will only use this user for ssh connection to the production server from CircleCI.

Head back to your local machine and create an ssh key pair which our new user will be using to connect to the server.

Make sure not to add a passphrase when prompted, or else CircleCI will not decrypt it.

$ cd
$ ssh-keygen -m PEM -t rsa -f .ssh/circleci

We cd to the home directory on the local machine. The -f will ensure that we create a key pair named circleci and circleci.pub

We are specifying the name circleci to avoid overwriting your other keys that might be existing.

Next, we copy the public key (circleci.pub).

$ cat ~/.ssh/circleci.pub

Select and copy the entire content, so we can register it on our production server.
So, ssh to your server again if you had closed the previous session and create a .ssh directory for the circleci user

$ sudo mkdir /home/circleci/.ssh

Then paste the public key into a file named authorized_key inside the .ssh directory

$ sudo nano /home/circleci/.ssh/authorized_keys

Change directory ownership and permissions for /home/circleci so that our user doesn't run into permission issues during the deployment

$ sudo chown -R circleci:circleci /home/circleci

also change the directory permission to allow all other users and groups read and execute access. This is important for when we are serving our static assets after deployment, but we will explain more about this later, for now just execute this command.

$ su circleci
$ chmod +R 755 /home/circleci

We can now test whether we can ssh to the server from our local machine. Remember our private key in our local machine? that's what we will use here.

ssh circleci@your_server_ip -i ~/.ssh/circleci

If you are able to log-in, it means that the ssh connection was successful and we can proceed.

Setting up the CircleCI account

Now you need to create a CircleCI here if you haven't already. You can sign up with your GitHub or Bitbucket account or your email.

If you sign up with GitHub or Bitbucket, all your account repos will be auto-imported. Otherwise, you'll have to do it manually (not a big deal).

Now head on to the CircleCI projects dashboard and click Setup Project button on the project you want to deploy with CircleCI.

Go to your local machine terminal again and copy the content of the circleci private key we generated earlier by running:

$ cat ~/.ssh/circleci

Back to the CircleCI project setup page, go to SSH KEY from the sidebar menu. Scroll down to the section Additional SSH keys and paste your key there. The hostname can be left blank. Remember to save it - this will allow CircleCI to ssh to the remote server.

Next, we set up some environmental variables required by CircelCI to perform its task. Some of the env variables are Host IP (host server IP), SSH User and App/release Name.

On the CircleCI project set-up page, locate Environmental Variables from the sidebar menu and add your .env variables there.

HOST_IP:********
APP_NAME: my_app
SSH_USER: circleci

Adding a CircleCI config.yml file

Let's write the test, build and deploy pipelines for our project. To do this, go to your Phoenix project on your local machine. Create .circleci folder in the root directory of your project. Then create a file named config.yml .

In our config.yml file, we start by telling CircleCI which images to use for our build machine.

# Elixir CircleCI 2.0 configuration file
# See: https://circleci.com/docs/language-elixir/
version: 2

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/configuration-reference/#jobs
jobs:
  build:
    # Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
    # See: https://circleci.com/docs/configuration-reference/#docker-machine-macos-windows-executor
    docker:
      # specify the version here 
      - image: cimg/elixir:1.14.4
      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # specify postgres image and postgres user for test purposes
      # documented at https://circleci.com/docs/circleci-images/ circleci/postgres:9.6
      - image: cimg/postgres:14.8
        environment:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: password

Here, I'm using CircleCI 2.0 configuration file. Next, I set up the jobs for our workflow.
For the build environment, we need to have Elixir and Postgres. CircleCI achieves this by using docker images. In my case, I'm using CircleCI convenience images (cimg). CircelCI will start a container with Elixir v1.14.4 and another one with Postgres v14.8.

You can use your preferred versions, but make sure to check whether those docker images exist.

I have also set up the default Postgres user and password, these will be used for the test pipeline. Make sure not to expose sensitive data in this file as it will be pushed to your git.

Next, we need to specify our app version and our working directory.

# set up version - optional but must match the app version configured nextpay-
environment: 
  - APP_VERSION: "0.1.0"

working_directory: ~/repo
    # Add steps to the job
    # See: https://circleci.com/docs/configuration-reference/#steps

Next, we add steps to our workflow. These steps will be executed in order one after another. In my case, I'll set up the build and deploy steps. You'll need to add more steps to perform your test. The test steps should run before the build to make sure that pushed code meet pass your test before deploying to the production server.

steps:
   - checkout

   # specify any bash command here prefixed with `run: `
   # Setup
   - run: mix local.hex --force
   - run: mix local.rebar

   # Testing Steps goes here

First, we check out our code and then we install hex locally and rebar.

- run:
   name: Install some necessary deps like rsync, nodejs, npm on the server for this case rsync will be installed
   command: sudo apt-get update && sudo apt-get install rsync

Here, I'm instructing CircleCI to install the necessary packages needed for our build and deploy task. These dependencies may include Nodejs, npm, and rsync - file copying tool. But I just want to install rsync as the rest is taken care of by the CircleCI convenience images.

Next, we fetch all dependencies, compile the app for production and deploy our static assets. This is done in the three steps below:

 # get dependencies and compile
 - run:
    name: Fetch mix dependencies
    command: MIX_ENV=prod mix deps.get --only prod
 - run: 
    name: Compile App
    command: MIX_ENV=prod mix compile
 # deploy static assets - css, js  
 - run:
    name: Deploy Static Files
    command: MIX_ENV=prod mix assets.deploy

Finally, we build our release and copy the release files to production server via ssh.
The variables used are the ones we did set up earlier under CircleCI environment variables.

# Build the Release
- run: 
    name: Build APP Release
    command: MIX_ENV=prod mix release

# Deploy to production server
- deploy: 
    name: Deploy to Production
    command: |
      ssh-keyscan -H $HOST_IP >> ~/.ssh/known_hosts;
      rsync -ravhz --stats --progress ./_build/prod/rel/$SSH_USER $SSH_USER@$HOST_IP:apps;
# post deploy step - stop and start the server
- deploy: 
    name: Post Deploy and App restart on production
    command: |
      chmod +x ./.circleci/post-deploy.sh
      ./.circleci/post-deploy.sh

You'll notice that the last step involves running a post-deploy.sh file. This file is the one that contains script to start and stop our app in the production environment. So create a file in your project under the .circleci folder we created earlier. Name it post-deploy.sh . The file looks like this:

#!/bin/bash

ssh $APP_USER@$APP_IP <<ENDSSH
    echo "switching directory to ./apps/$APP_NAME";
    cd ./apps/$APP_NAME;
    date >> deploy-log.txt;

    echo "Stopping application...";
    ./bin/$APP_NAME stop >> deploy-log.txt || true;
    echo "Exiting";
ENDSSH

ssh $APP_USER@$APP_IP <<ENDSSH
    echo "Changing directory into ./apps..";
    cd ./apps;

    echo "Getting environment variables ..";
    source .env;

    echo "$APP_NAME";
    date >> deploy-log.txt;

    echo "Starting application...";
    ./bin/$APP_NAME daemon >> deploy-log.txt

    echo "Finished";
ENDSSH

Here we are stopping the previously running application daemon and starting it again.

Our final CircleCI config.yml should look like this:

# Elixir CircleCI 2.0 configuration file
# See: https://circleci.com/docs/language-elixir/
version: 2

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/configuration-reference/#jobs
jobs:
  build:
    # Specify the execution environment. You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
    # See: https://circleci.com/docs/configuration-reference/#docker-machine-macos-windows-executor
    docker:
      # specify the version here 
      - image: cimg/elixir:1.14.4
      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # specify postgres image and postgres user for test purposes
      # documented at https://circleci.com/docs/circleci-images/ circleci/postgres:9.6
      - image: cimg/postgres:14.8
        environment:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: password

    # set up version - optional but must match the app version configured nextpay-
    environment: 
        - APP_VERSION: "0.1.0"

    working_directory: ~/repo
    # Add steps to the job
    # See: https://circleci.com/docs/configuration-reference/#steps
    steps:
      - checkout

      # specify any bash command here prefixed with `run: `
      - run: MIX_ENV=prod mix local.hex --force
      - run: MIX_ENV=prod mix local.rebar

      # Testing Steps goes here

      # install necessary build dependencies
      - run:
          name: Install some necessary deps like rsync, nodejs, npm on the server for this case rsync will be installed
          command: sudo apt-get update && sudo apt-get install rsync

      # get mix dependencies and compile
      - run:
          name: Fetch mix dependencies
          command: MIX_ENV=prod mix deps.get --only prod
      - run: 
          name: Compile App
          command: MIX_ENV=prod mix compile
      # Deploy static files - css, js
      - run:
          name: Deploy Static Files
          command: MIX_ENV=prod mix assets.deploy

      # Build the Release
      - run: 
          name: Build APP Release
          command: MIX_ENV=prod mix release

      # Deploy to production server
      - deploy: 
          name: Deploy to Production
          command: |
            ssh-keyscan -H $APP_IP >> ~/.ssh/known_hosts;
            rsync -ravhz --stats --progress ./_build/prod/rel/$APP_NAME $APP_USER@$APP_IP:apps;
      # post deploy step - stop and start the server
      - deploy: 
          name: Post Deploy and App restart on production
          command: |
            chmod +x ./.circleci/post-deploy.sh
            ./.circleci/post-deploy.sh

It's now time for you to commit and push your code to your GitHub or Bitbucket repo.
But there's one more thing we need to do.

Our circleci user needs to authenticate with GitHub or Bitbucket to run tasks such as git pull.

We will create an ssh key pair - yes another ssh key pair - for our user to authenticate against Github. For Bitbucket users you don't need to do this.

So, ssh to your production server as circleci .

$ ssh circleci@your_server_ip -i ~/.ssh/circleci

Create the key pair with no passphrase

$ ssh-keygen -t rsa

Then output the public key and copy

$ cat ~/.ssh/id_rsa.pub

Copy the output and then head on to your CircleCI dashboard. Click the Project Settings button next to your repo and go to SSH Keys from the sidebar. Under Checkout SSH Keys click on Add Deploy Key to add the copied key. The key name should be your desired name, then paste the copied key under Key field then clicks Add Key to save. For Bitbucket users, this is done automatically thus no action is needed.

Now that your circleci user has access to your GitHub or Bitbucket, it's time to commit and push your code to your repo.

CircleCI will now pull your code to the build machine, build and deploy the release to your production server, and start the app on production.

Voila! Thank you for taking the time to read this kinda long post. Let me know if you have a question or need help in the comments.

Build and Deploy Elixir App with Releases and CircleCI

Requirements:

Setting up the production server

Setting up the CircleCI account

Adding a CircleCI config.yml file