Using Sidekiq with Flightcontrol

Running Sidekiq (or any other background worker) with Flightcontrol is straight forward. You will need to add Sidekiq into your Ruby project first, but then you can deploy Sidekiq as a worker that talks to an AWS ElastiCache Redis instance.

Understanding how Flightcontrol Deploys Workers

Flightcontrol supports several different Service Types. The one we are interested in is the worker service type. A worker service type is useful for running a process that executes background jobs. Typically, this is deployed next to a web service that enqueues jobs for the worker to process.

You will want to deploy your web service and your worker service in the same Flightcontrol environment. This way, they share a Redis instance. The web service can use the Redis instance for caching, and the worker can use the Redis instance for its job queue.

Rails and Sidekiq

Rails has built in support for background jobs with the ActiveJob framework. This gives you some flexibility to swap out the background job implementation later if you need to.

You can also use Sidekiq directly for background jobs.

The important thing for deploying Sidekiq with Flightcontrol is to use the right startCommand for your worker service. For Sidekiq, you will want to use bundle exec sidekiq.

If you were using another background job framework, you would use the appropriate command to start that framework.

Configuring ElastiCache

Flightcontrol supports deploying ElastiCache Redis instances within Flightcontrol environments. You can configure your Sidekiq worker to use this instance for its job queue, using the REDIS_URL environment variable.

ElastiCache has many supported node types - AWS maintains a List of ElastiCache Node Types to choose from.

You can also choose which version of Redis to use. This is important if you are on an older version of Sidekiq - you’ll want to make sure that the version of Sidekiq you use is compatible with the version of Redis you choose.

Example flightcontrol.json Configuration

Below is an example flightcontrol.json file you could use as the basis for your own configuration. This example deploys a Rails application with a Sidekiq worker. The Sidekiq worker is configured to use an ElastiCache Redis instance. The Rails application has access to the same Redis instance for caching.

For your own application, you will probably want to adjust the number of instances. You’ll also want to size your application properly with the RDS and ElastiCache instance types and the CPU and memory specified for the web service and worker.

{
  "$schema": "https://app.flightcontrol.dev/schema.json",
  "environments": [
    {
      "id": "production",
      "name": "Production",
      "region": "us-west-2",
      "source": {
        "branch": "main"
      },
      "services": [
        {
          "id": "rails-server",
          "name": "Rails Server",
          "type": "web",
          "target": {
            "type": "fargate"
          },
          "buildType": "nixpacks",
          "ci": {
            "type": "ec2"
          },
          "cpu": 0.25,
          "memory": 0.5,
          "minInstances": 1,
          "maxInstances": 1,
          "buildCommand": "bundle exec rails db:migrate && bundle exec rake assets:precompile",
          "envVariables": {
            "RAILS_ENV": "production",
            "DATABASE_URL": {
              "fromService": {
                "id": "db",
                "value": "dbConnectionString"
              }
            }
          }
        },
        {
          "id": "sidekiq-worker",
          "name": "Sidekiq Worker",
          "type": "worker",
          "target": {
            "type": "fargate"
          },
          "buildType": "nixpacks",
          "ci": {
            "type": "ec2"
          },
          "cpu": 0.25,
          "memory": 0.5,
          "minInstances": 1,
          "maxInstances": 1,
          "startCommand": "bundle exec sidekiq"
        },
        {
          "id": "db",
          "name": "Database",
          "type": "rds",
          "engine": "postgres",
          "engineVersion": "12",
          "instanceSize": "db.t4g.micro",
          "storage": 20,
          "private": false
        },
        {
          "id": "redis",
          "name": "Redis",
          "type": "elasticache",
          "engine": "redis",
          "engineVersion": "7.0",
          "instanceSize": "cache.t4g.small",
          "connectionStringEnvVarName": "REDIS_URL"
        }
      ]
    }
  ]
}