Motivation

Recently, we were approached by one of our clients seeking our and assistance with their Django application. The client was hosting their Django application on a Kubernetes cluster in AWS. We needed to integrate certain operations into their CI/CD pipeline, specifically running Django migrations that required a database connection. The challenge was that the database was hosted on an on-premise server, protected by a firewall with IP-whitelisting to secure data exchange between trusted sources. To address this, we decided to execute the job on an EC2 instance in AWS. This instance had a public IP, which we were able to add to our firewall's whitelist, thus ensuring a secure and reliable connection to the database for the necessary operations.

In this article, I'll guide you through the process of setting up self-hosted GitLab runners on AWS EC2. But before we dive into the setup, let's first take a moment to understand what GitLab runners are and their role in streamlining your development workflow. This foundational knowledge will help in appreciating the significance of the steps that follow.

Gitlab runners

GitLab Runners are tools used with GitLab to help automatically test and deploy your code. Think of them as assistants that carry out tasks you define, like checking if your code works or updating your app. You can set up these runners on different machines, like your own computer or a server, and they can handle different jobs at the same time. Gitlab offers two kinds of runners:

• Shared runners
• Self-hosted runners

Shared runners in GitLab are accessible to all its users and are managed by GitLab. On the other hand, self-hosted runners are those you set up and manage on your own. You can host them on an on-premise server or use public cloud services like AWS, Azure, or GCP. In this article, we'll focus on setting up a GitLab runner using AWS EC2 instances. Although we're using AWS as our example, it's worth noting that the setup process is quite similar if you choose to use Azure or GCP.

Set up EC2 instance

The first step in our process is to create an EC2 instance on AWS. For this article, I'll demonstrate how to do this using the AWS console, though you could also use Terraform or the AWS CLI. Let's name our instance 'my-gitlab-runner'. The specific configuration of the EC2 instance isn't too critical for our purposes. However, since we need a public IP to whitelist in our database firewall, I'll make sure to enable the 'Auto-assign public IP' option during setup. We'll use the Amazon Linux 2023 AMI for the instance. Additionally, it's important to configure the instance to allow inbound traffic on port 22, as we'll need SSH access to perform certain configurations on the instance.

Set up Gitlab runner

First thing we need to do is to set up our Gitlab runner in our Gitlab project. Go to your Gitlab project, on the left bar press Settings -> CI/CD. Here, on the Runners tab, press Expand. Then, press New project runner.

Now, you'll be promted some configurations for the runner.

Here I specify that I'll be using a Linux machine to host my runner. I also give it the tag ec2-runner. This is important as this is what we'll be referencing from our Gitlab runner file later. After you've hit Create runner, there will be a token displayed. You need to write this down for later.

Register Runner

Now that our runner is configured in Gitlab, we need to register it on our EC2 instance.

SSH into you EC2 instance by running the following command

$ ssh ec2-user@<public-ip> -i your-key.pem

Then, we need to add the Gitlab runner to our repository

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

Next, we can install the package by running

sudo yum install gitlab-runner

Finally, we need to register the runner

sudo gitlab-runner register  --url https://gitlab.com  --token <your-gitlab-runner-token>

Here we need to add our token that we got after creating our project runner in Gitlab earlier. The --url references where we are hosting our project. 

This will prompt you to fill in,

• Gitlab instance URL, leave as default when using Gitlab.
• Name for the runner, this is an identifier for our runner. I'll call this ec2-gitlab-runner.
• Executor, I chose shell as my executor. Shell is recommended unless you have a specific need for e.g Docker or Kubernetes. Please refer to Gitlab Documentation for more information on choosing executor. 

Now our Gitlab runner is registered and ready to use.

Test our runner

Now, we can test our Gitlab runner by creating a .gitlab-ci.yaml file in our Gitlab project.

build-self-hosted-runner:
  stage: build
  tags:
    - ec2-runner
  script:
    - curl httpbin.org/ip
  when: manual

In this step, we'll set up a job in the usual manner. However, it's important to pay attention to the tags argument. The tag you assign here must match the one you specified while setting up your runner. This functionality is quite useful as it enables different jobs to run on distinct runners, all configured within the same .gitlab-ci.yaml file. For our scenario, we had two different jobs: one ran on GitLab's shared runners and the other, as described above, was designated to run on our self-hosted EC2 GitLab runner. When this job is executed, you should see something like the this.

As we can see, the EC2 instance public IP is printed out (parts of image has been blurred for privacy reasons). This confirms that the job was infact run on the EC2 instance.

Summary

In this article, I guided you through setting up a self-hosted GitLab Runner on an AWS EC2 instance. The process began with creating an AWS EC2 instance, 'my-gitlab-runner', including enabling 'Auto-assign public IP' and configuring Amazon Linux 2023 AMI with SSH access.

Next, I demonstrated how to set up a GitLab runner in a GitLab project and how to register this runner on the EC2 instance. This involved SSH into the instance, installing the GitLab runner package, and registering the runner with a specific token and URL.

Finally, I provided an example of testing the runner with a .gitlab-ci.yaml file in the GitLab project, highlighting the importance of matching the tags in the job configuration with the runner setup. The successful execution of this test confirmed the job was run on the EC2 instance.

This article aims to provide a comprehensive guide to enhancing CI/CD workflows with GitLab Runners on AWS EC2.

Hopefully you found this article useful!