Running Flower on AWS ECS for Celery Monitoring

Edit: I would not recommend using Flower in production. I've discovered that it scales poorly with celery brokers that handle a lot (>=1k/hr) of jobs. I have left this guide here, should you wish to deploy Flower to ECS regardless.

Celery is the most popular distributed queue library for Python, but doesn't have much in the way of monitoring features. Therefore Flower has become the canonical tool for Celery monitoring. Flower is designed as a package that can be run in a Python environment, or, as I'll show you here, can be containerised and run as it's own service.

Flower, like Celery, has it's quirks. I did look for alternatives, and come across Leek, an is open source, celery monitoring tool. Unlike Flower, it supports multiple brokers, and what's more, it's designed to be deployed as a full-stack containerised app. Unfortunately Leek is still in it's early days, and I encountered several bugs. Hopefully with further development it will become a solid competitor to Flower, as it looks very promising.

Below is a guide for running Flower in production, using AWS Elastic Container Service and Fargate. Though I'll focus on ECS, the advice might be useful for running on any managed container service.

Creating your Flower task definition

The task definition is a JSON file that describes the images to use for your tasks, as well as various options for launch.

Once in the ECS console, create a new task definition:

Infrastructure requirements

• Launch type: AWS Fargate (you could use EC2, but I have opted for Fargate for simplicity).
• Operating system: Linux/X86_64 - I haven't tested Flower on an ARM Architecture, though it might work just fine.
• Task size: CPU: .5vCPU, 1GB memory. This has been sufficient for me, YMMV.
• Task role: You can make a basic role, it shouldn't need much other than maybe access to Cloudformation for pushing logs.
• Execution role: You'll need to make an execution role - it can just have AmazonECSTaskExecutionRolePolicy attached.

Container 1 - the Flower container

Name your container, and make it essential. You can make the image URI point to my own image, which is a slight modification of the Flower 2.0.1 Dockerfile: amaitu/flower:0.0.4. I made this modification as I had issues with the user set up in the official Dockerfile.

Set your container port to be 5555. This is the default port for flower, and you can point a load balancer to this port.

Environment variables

• CELERY_BROKER_URL
  • your-redis-uri
  • e.g. redis://xxxxx.xxxxx.ng.0001.euw2.cache.amazonaws.com:6379/0
• CELERY_RESULT_BACKEND
  • your-redis-uri
  • e.g. redis://xxxxx.xxxxx.ng.0001.euw2.cache.amazonaws.com:6379/0
• FLOWER_DEBUG
  • false
  • Can be used for debugging, but will print sensitive information, so turn off for production.
• PURGE_OFFLINE_WORKERS
  • 1200
  • This is the duration in seconds after which offline workers will no longer show on the Flower dashboard. This is optional, but for a horizontally scaled queue system where queue workers are ephemeral, you may want it to be short.
• FLOWER_PERSISTENT
  • true
  • This means Flower saves state to disk, and so your task history will be available between restarts. If you have this enabled, see persisting state below.

Persisting state - EFS

Flower wasn't designed to be a stateless containerised application: for instance, if you want data to be persisted between container runs, it needs a disk to write to (😢). Flower uses shelve for it's database - object persistence that's part of the standard library. For running on ECS, I opted for using Amazon Elastic File System (EFS). These filesystems are hands-off, 'serverless' and can be mounted to containers.

Ensure you have set up a file system, then in your task definition, click 'Add volume'.

Don't forget when you set up your file system, to create mount target(s) with a security group. Ensure this security group allows traffic on port 2049, for EFS.

Choose your File system ID, and make the volume type 'EFS'. Make the root directory '/'.

Be sure to add a container mount point, where the source volume is EFS, the container is your Flower container defined above, and your container path is '/flower' if you are using amaitu/flower, or '/data/' if you are using the mher/flower image.

Adding a Flower container health-check

I'd recommend adding a container health-check. As the base image for Flower doesn't have curl, I've found that a connection in Python using socket works:

CMD,python,-c,import socket; s = socket.socket(socket.AF_INET, socket.SOCK_STREAM); s.connect(('localhost', 5555))

Service creation

When you come to create your Flower service, that you set your max running tasks percentage to be 100. This way, during deployments, ECS won't allow >1 Flower container to run. Running more than one at once could have unindented consequences, given this app saves the database to disk.

You can now start your service! Flower should boot in about 20 seconds

Other resources to create

How you connect to this service will depend on your particular environment. However for a typical web service, you may want to create the following other resources:

• A log group: for monitoring logs from Flower;
• A load balancer: to route internet-facing traffic to our service;
• A certificate: for handling incoming https connections;
• A security group, to allow incoming traffic to the load balancer.