Message Queue Framework with RabbitMQ – publisher, consumer, DDEV config

Operations like sending emails, syncing with an ERP, or recalculating indexes do not have to block an HTTP request. Magento 2 has a built-in Message Queue Framework that lets you push such tasks to an asynchronous queue. I show how it works with RabbitMQ and how to write your own publisher and consumer.

What is the Message Queue Framework in Magento 2?

MQF is an abstraction layer over message brokers. Magento supports two backends:

• MySQL – the default, zero configuration, sufficient for smaller deployments
• RabbitMQ – recommended for production environments, supports routing, retry, dead letter queues

Architecture: Publisher (sends message), Queue (stores it), Consumer (processes it).

Module configuration – XML files

Define the message topic in etc/communication.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="vendor.module.order.export"
           request="Vendor\Module\Api\Data\OrderExportRequestInterface"/>
</config>

Queue and binding configuration in etc/queue_topology.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/topology.xsd">
    <exchange name="magento" type="topic" connection="amqp">
        <binding id="orderExportBinding"
                 topic="vendor.module.order.export"
                 destinationType="queue"
                 destination="vendor.order.export"/>
    </exchange>
</config>

Register the consumer in etc/queue_consumer.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="vendor.order.export.consumer"
              queue="vendor.order.export"
              connection="amqp"
              handler="Vendor\Module\Model\Consumer\OrderExportConsumer::process"
              maxMessages="100"/>
</config>

Publisher – sending messages to the queue

<?php

namespace Vendor\Module\Model;

use Magento\Framework\MessageQueue\PublisherInterface;
use Vendor\Module\Api\Data\OrderExportRequestInterfaceFactory;

class OrderExportPublisher
{
    private const TOPIC = 'vendor.module.order.export';

    public function __construct(
        private PublisherInterface $publisher,
        private OrderExportRequestInterfaceFactory $requestFactory
    ) {}

    public function publish(int $orderId): void
    {
        $request = $this->requestFactory->create();
        $request->setOrderId($orderId);

        // Message goes into the queue - HTTP request ends immediately
        $this->publisher->publish(self::TOPIC, $request);
    }
}

Consumer – processing messages

<?php

namespace Vendor\Module\Model\Consumer;

use Vendor\Module\Api\Data\OrderExportRequestInterface;

class OrderExportConsumer
{
    public function __construct(
        private \Psr\Log\LoggerInterface $logger,
        private \Vendor\Module\Model\ErpExporter $erpExporter
    ) {}

    public function process(OrderExportRequestInterface $request): void
    {
        $orderId = $request->getOrderId();

        try {
            $this->erpExporter->export($orderId);
            $this->logger->info('Order exported successfully', ['order_id' => $orderId]);
        } catch (\Exception $e) {
            // Exception = message goes to dead letter queue (with RabbitMQ)
            $this->logger->error('Export failed', [
                'order_id' => $orderId,
                'error'    => $e->getMessage(),
            ]);
            throw $e; // pass on - Magento handles retry
        }
    }
}

Starting the consumer

# Start manually (for testing)
bin/magento queue:consumers:start vendor.order.export.consumer

# With message limit
bin/magento queue:consumers:start vendor.order.export.consumer --max-messages=500

# On production - via supervisor
# [program:magento_order_export]
# command=/var/www/magento/bin/magento queue:consumers:start vendor.order.export.consumer
# autostart=true
# autorestart=true

RabbitMQ in DDEV

# .ddev/docker-compose.rabbitmq.yaml
version: '3.6'
services:
  rabbitmq:
    image: rabbitmq:3.8-management
    environment:
      RABBITMQ_DEFAULT_USER: magento
      RABBITMQ_DEFAULT_PASS: magento
    ports:
      - "5672"
      - "15672:15672" # management panel
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}

The RabbitMQ management panel is available at http://localhost:15672 – you can monitor queues, message counts and dead letter messages there.

Summary

The Message Queue Framework in Magento 2 lets you move time-consuming operations outside the request-response cycle. The XML configuration looks verbose but follows a repeatable pattern. RabbitMQ with retry and dead letter queues is the right choice for production environments handling large order volumes.