Technology RadarTechnology Radar

RabbitMQ

What is a queue?

A queue, is a data structure for the linear collection of elements that follows the First-In, First-Out (FIFO) principle. Queues are commonly used in scenarios where order matters.

What is a work queue?

A work queue on the other hand, extends the basic concept. With work queues you have the work queue itself, producers that add jobs to it, and consumers (workers) which retrieve jobs from the work queue and process them.

Why use work queues?

This allows to asynchronously process long-running tasks that would otherwise block a system. A good example is when a video is transcoded whenever a new video asset is uploaded.

As workers only take on a job whenever they are able to do, the work queue ensures we don't overwhelm the computing systems the workers run on. On the other hand workers can be scaled to a number that would utilize the system properly.

If you have a workload that can be split up, you're architecture will benefit from parallel processing. Workload will be distributed to all workers consuming from a queue.

Outside distribution, work queues help in decoupling systems. A producer doesn't need to know how and by whom a job is going to be processed. You can use a different language or framework for the consumer.

As a downside work queues don't provide a good mechanism for feedback. Work queues usually don't provide means to provide feedback to the producer, i.e. "video transcoded".

RabbitMQ

RabbitMQ positions itself more generally as a message broker, through which it is possible among other things to provide work queues. Work queues play a major role in the messaging technology space.

Learn more about RabbitMQ

Updates

Adopt

At Factorial we tend to use RabbitMQ's work queue features only. These includes durable queues with the persistence of jobs, when the system goes offline, priorities, TTL and requeuing support.

Trial

Within Website Standards we are trying out RabbitMQ as we evolve the platform.