Introduction to RabbitMQ Architecture

This blog post explains what is RabbitMQ and message queueing and also gives brief understanding of architecture of RabbitMQ.

What is RabbitMQ

RabbitMQ is a open source message broker  also known as a message-queueing software or queue manager. It receives and delivers message from and to applications.

It is software where queues are defined, to which applications connect in order to transfer a message or receive a message

RabbitMQ is lightweight and easy to deploy on premises and in the cloud. It supports multiple messaging protocols. RabbitMQ can be deployed in distributed and federated configurations to meet high-scale, high-availability requirements.

RabbitMQ is built based on AMQP (Advanced Message Queuing Protocol) protocol.

AMQP (Advanced Message Queuing Protocol) is a messaging protocol that enables conforming client applications to communicate with conforming messaging middleware brokers.

When and why should you use RabbitMQ?

We have to know about Synchronous and Asynchronous request processing.

When user interacting with applications , he expects response immediately. The process is called synchronous process, web application moves to next request only after responding to current request.

As the number of users increases , the application might face scaling issues to process all the requests in real time.

There will be some scenarios which need not processed in real-time. For those requests application accepts the request from user but sends response with delayed time.

For example

If user is trying to export bank statement in PDF format. Application can accept the request and send the response in a mail.

In above scenarios, application needs to store the request in some place so that it can process the request at a later time.

Message queues are used to store the requests for which applications decided to respond asynchronously.

Message queueing enables web servers to react to requests rapidly rather than being forced to conduct resource-intensive operations on the fly, which may cause response time to be delayed. Message queueing is also good when you want to distribute a message to multiple consumers or to balance loads between workers.

RabbitMQ Architecture

Let’s look at the flow of messages/request and how it processed with the help of diagram

1. The user/client sends a PDF creation request to the web application.
2. The web application (the producer) sends a message to RabbitMQ that includes data from the request such as name , email, account number and date range.
3. An exchange accepts the messages from the producer and routes them to correct message queues for PDF creation.
4. The consumer (PDF processing worker) receives the message and starts processing the message to produce the PDF.

MESSAGES

The message contains the information to complete the request. It could contain plain text data, image or document. For example, in case of bank statement it contains account number, date range and email.

PRODUCER

Producer is a program which generates the messages ( requests) and sends it to RabbitMQ.

CONSUMER

Consumer is a application which consumes messages from Queue and process them.

Applications consume messages 2 ways.

• Subscription
• Polling

In Subscription method, messages are pushed to consumer.In this method applications have to indicate interest in consuming messages from a particular queue(i.e. subscribe to a queue).

It is possible to register more than one consumer per queue or an exclusive consumer (excludes all other consumers from the queue while it is consuming).

A consumer tag is an identification assigned to each customer (subscription). It is possible to utilize it to unsubscribe from messages. Consumer tags are just strings.

Subscription method is efficient method.

In Polling method consumer will poll the queue at regular interval for messages.

Polling is inefficient method.

EXCHANGES

Messages are not published directly to a queue; instead, the producer publishes messages to an exchange. An exchange is in charge of routing messages to different queues using bindings and routing keys. A binding connects a queue and an exchange.

You can think of Exchanges like post offices if you are sending paper based mail .

TYPES OF EXCHANGES

Direct Exchange

A direct exchange delivers messages to queues based on a message routing key.

The routing key is a message attribute added to the message header by the producer. Routing key is like an “address” that the exchange is using to decide how to route the message. A message goes to the queue(s) with the binding key that exactly matches the routing key of the message.

The direct exchange type is useful to distinguish messages published to the same exchange using a simple string identifier.

In above diagram, a message with routing key Event_A is sent to the exchange Queue A. The messages with routing key Event_B is routed to QUEUE B because the routing key (EVENT_B) matches the binding key.

If the message routing key does not match any binding key, the message is discarded

Default exchange

The default exchange is a pre-declared direct exchange with no name, usually referred by an empty string. When you use default exchange, your message is delivered to the queue with a name equal to the routing key of the message. Every queue is automatically bound to the default exchange with a routing key which is the same as the queue name.

Topic Exchange

Topic exchanges route messages to queues based on wildcard matches between the routing key and the routing pattern, which is specified by the queue binding. Messages are routed to one or many queues based on a matching between a message routing key and matching pattern.

The routing key must be a list of words, delimited by a period (.).

routing key can contain 2 special keys * or #

# – indicates a match of zero or more words

* – matches a word in a specific position of the routing key

Examples are report.monthly and report.*.weekly which in this case identifies reports that are set up for a bank.

The routing patterns may contain an asterisk (“*”) to match a word in a specific position of the routing key (e.g., a routing pattern of “report.*.*.weekly” only match routing keys where the first word is “report” and the fourth word is “weekly”).

A pound symbol (“#”) indicates a match of zero or more words (e.g., a routing pattern of “reports.monthly.#” matches any routing keys beginning with “reports.monthly”).

If producer sends a message with routing key “report.monthly” , it will match the routing key of

• report.monthly.#
• report.#

So message is routed to Queue A and Queue C

If producer sends a message with routing key “report.personalaccount.weekly” , it will match the routing key of

• report.*.weekly
• report.#

So message is routed to Queue B and Queue C

The consumers indicate which topics they are interested in (like subscribing to a feed for an individual tag). The consumer creates a queue and sets up a binding with a given routing pattern to the exchange. All messages with a routing key that match the routing pattern are routed to the queue and stay there until the consumer consumes the message.

Fanout Exchange

A fanout exchange copies and routes a received message to all queues that are bound to it regardless of routing keys or pattern matching . The keys provided will simply be ignored.

Fanout exchanges can be useful when the same message needs to be sent to one or more queues with consumers who may process the same message in different ways.

The above image (Fanout Exchange) shows an example where a message received by the exchange is copied and routed to all three queues bound to the exchange. It could be political ,sports or weather updates that should be sent out to each connected

Headers Exchange

A headers exchange routes messages based on arguments containing in headers and optional values. Headers exchanges are very similar to topic exchanges, but routes messages based on header values instead of routing keys. A message matches if the value of the header equals the value specified upon binding.

A special argument named “x-match” which specifies criteria matching, added in the binding between exchange and queue, specifies

• if all headers must match or just one.
• Either any common header between the message and the binding count as a match
• or all the headers referenced in the binding need to be present in the message for it to match.

The “x-match” property can have two different values:

“any” or “all”

all – is the default value. A value of “all” means all header pairs (key, value) must match

any – means at least one of the header pairs must match.

Headers can contain wider range of data types, integer string or boolean. The headers exchange type (used with the binding argument “any”) is useful for directing messages which contain a subset of known (unordered) criteria.

Dead Letter Exchange

If no matching queue can be found for the message, the message is silently dropped. RabbitMQ provides an AMQP extension known as the “Dead Letter Exchange”, which provides the functionality to capture messages that are not deliverable.

QUEUES

Queues store messages that are consumed by applications. 

Bindings

Bindings are rules that exchanges use (among other things) to route messages to queues. Some exchange types may employ an optional routing key property on bindings. The routing key’s function is to route specific messages published to an exchange to the bound queue. To put it another way, the routing key functions as a filter.

Having this layer of indirection enables routing possibilities that are impossible or extremely difficult to accomplish when publishing directly to queues, as well as eliminating some of the repeated work that application developers must undertake.

If a message cannot be routed to any queue (for example, because there are no bindings for the exchange to which it was published), it is discarded or returned to the publisher, based on the message properties given by the publisher.

Difference between Routing Keys and Binding Keys

routing keys are on messages

binding keys are on bindings(routes) between exchange and queues,

Exchanges compare a messages routing key to each route’s binding key to determine if the message should be sent to the queue on that route.

You might be interested in