Configuring multi-threaded tests on Heroku CI for Ruby on Rails

What is Heroku CI?

Heroku CI is continuous integration service provided by Heroku.
It does not require complicated setup that can be tedious for users and is easy to setup even by user inexperienced in DevOps duties (like myself).
One of the biggest benefits is that you can run as many branches as you like simultaneously.

First steps

For first few steps Heroku provides very good documentation. I want to sum it up just to make any reader who may stumble upon this article reassured that it was done correctly.

Firstly create a pipeline, and add your app to it. If you don’t add any app to the pipeline, pipeline will not be shown in the future. So if you have any objections against doing it, or just want to add app later, you’ll have to repeat whole process.

Then you have to connect Heroku with your GitHub repo, and your app is fully connected to pipeline

Lastly turn on Heroku CI. It is separate section in your pipeline on settings tab.

Schema — app.json

App.json is where all magic happens. It is Heroku configuration file responsible for setting up your environment.
In the context of CI this is your most important file.
If you will write this file correctly your test suite will be running.
On the other hand if something is wrong with test suite in 90% of cases it is the app.json file that is responsible for the failure.

So let’s dive into it’s structure:

{
  "environments": {
    "test": {
    }
  }
}

First of all declare all things you will need for tests under “test:” key.
Then declare what other resources you will need:

"addons": [
  {
    "plan": "heroku-postgresql:in-dyno"
  },
  {
    "plan": "heroku-redis:in-dyno"
  }
],

In my case I needed PostgreSQL for main database, and Redis for Sidekiq tests.

You may also need to declare buildpacks:

"buildpacks": [
  {
    "url": "heroku/ruby"
  }
],

And don’t forget about environment variables:

"env": {
  "non_secret_env": "test",
  ...
}

Don’t put any crucial data there. If you are afraid of putting some env’s here there is also “Test run config vars” under Heroku CI tab in your pipeline settings.

If you are here just for the tips on starting normal Heroku CI your ride ends here. Just put in your app.json test setup and test commands, depending on your test suite and you are ready to go.

"scripts": {
  "test-setup": "bundle exec rake db:migrate",
  "test": "bundle exec rspec"
}

But for me it was not enough, so I created a script:

"scripts": {
  "test-setup": "chmod +x ./ci/run_tests.sh",
  "test": "./ci/run_tests.sh"
}

Why?
I had a Frontend Jest tests, and Backend RSpec tests. You cannot run them unless you run them both in the same shell script.

Main part of this article starts here.

This is where the fun begins. Fasten your seatbelts and be ready to accelerate your tests to overdrive.

At this point I’ve had a running test suite (as you should).
On project which I am doing in time of writing this article we had RSpec tests for Backend, and Jest for Frontend.
We were moving from Jenkins to Heroku CI for reasons that are not important from our point of view.
The only reason I mention this is the time of average run of the test suite.

It took about 30–40 minutes to run those tests, and we run Backend through parallel_tests gem.

On default Heroku CI configuration it took me about 25 minutes to run them all.

What is performance?

Performance can have many meanings and many people smarter then me gave us definitions of performance. For this context for me performance is:

MY TESTS HAVE TO GO FASTER

First of all here you have the “missing” part of my app.json:

"formation": {
  "test": {
    "quantity": 1,
    "size": "performance-l"
  }
},

This part requests performance-l dyno from Heroku.

Why?

At the moment of writing this article performance-l dyno is two times more expensive then default one and it is more then two times faster for my suite. I cannot guarantee it will happen for your test suite, but try it, maybe it is worth a shot. If not you can always go back to the default performance-m, or even lower.

The Gamechanger

So, do you remember I told you about parallel_test gem we used on Jenkins?
It needs database for every process. It means that we cannot use it unless we figure out how to create databases on Heroku CI that actually work.

What is wrong with just creating the database?

Nothing, it just does not work. But, why it doesn’t work?

For our convenience, Heroku provides us with DATABASE_URL, and this means all parallel tests that uses different databases are pointed to the one with URL.
Unfortunately for us we cannot simply override it with empty value as we need it for access to database.

So we need to create a role. Fortunately Heroku leaves trust mode turned on.
So we can just:

#run_tests.sh
unset DATABASE_URL
psql postgres -c "CREATE USER postgres SUPERUSER;"

Then just create databases (remember that you have one named postgres_buildpack already in game):

#run_tests.sh
...
psql postgres -c "ALTER DATABASE postgres_buildpack_db RENAME TO project_test;"
psql postgres -c "CREATE DATABASE project_test2 WITH TEMPLATE project_test"
...

And run your tests in the same script

#run_tests.sh
bundle exec parallel_rspec -n number_of_your_processes

In my case optimal number was 12. What is this number for you?

You can only determine it by tries and errors method.

Summary

For my project I was able to get down to average time about 7 minutes and 15 seconds, while running both FE and BE tests.

Thanks to Mariusz Błaszczak