Kafka, RabbitMQ, or NATS: A Billing Firm's Payout Queue Decision
Archi4 min read
The Queue That Failed an Audit#
The company sells subscription billing to 4,200 small merchants. Its payout queue ran on Redis lists and a cron worker. On a normal Tuesday it moved 600 events a minute; on the first of the month it hit 1,800. The system worked until it did not. A duplicate payout in January cost $18,400 in manual reversals and two weeks of support time. Then an audit asked for seven days of event replay by merchant ID. Redis lists could not do that without a separate archive. The vendor had 11 weeks to replace the queue.
The team shortlisted Apache Kafka, RabbitMQ, and NATS JetStream. All three are open source, all three persist messages, and all three can be bought as managed services. The differences showed up in replay, failover, and the cost of people who know the system.
Apache Kafka#
Kafka treats the queue as an append-only log. Replay is a config change: keep seven days of retention, reset a consumer group, read forward. For a billing audit, that is a direct fit. At the company's volume, three brokers and 2 TB of storage were enough. A managed quote came to $2,100 a month. Self-hosting on three 8 vCPU nodes landed near $1,400 a month before staff time.
The cost that did not appear on the quote was operations. Kafka's partition model, consumer group rebalances, and schema registry add work. The team had one engineer who had run Kafka at a previous job. The rest knew queues as a service. In a 10-day test, a Kafka cluster passed replay in four minutes and survived a broker kill in 31 seconds. Building a dead-letter view and a merchant-level replay tool took 38 engineer-hours.
RabbitMQ#
RabbitMQ was the familiar option. Two engineers had used it. The management UI made routing and dead-letter queues visible, and quorum queues handled the 1,800 events a minute without strain. A managed RabbitMQ quote was $1,100 a month. Failover in the test took eight seconds, the fastest of the three.
Replay was the problem. RabbitMQ can stream messages, but the audit rule was not a full log replay. It was a filtered replay by merchant ID and date range. The team could build that by writing every payout event to Postgres as an audit table, then republishing from there. That is a sound pattern. It also meant two systems to keep in sync. In the bake-off, a filtered replay took 19 minutes using a Postgres export and a re-publish job. The team estimated 22 engineer-hours to make it reliable.
NATS JetStream#
NATS JetStream looked like the smallest tool for the job. A three-node cluster can run on modest hardware, and subject-based routing fits a payout queue that is already organized by merchant and currency. A managed quote was $900 a month; self-hosting three small nodes was under $500. The test cluster replayed seven days of events in six minutes and failed over in 12 seconds.
The risk was the hiring pool and the tooling. JetStream has retention, consumer groups, and key-value stores, but fewer engineers have run it in production than Kafka or RabbitMQ. The team found no managed provider with the same audit exports and no internal schema registry. They would write more glue code. The estimate was 16 engineer-hours for setup, lower than the others, but with a higher unknown for year two.
The Decision#
The company picked RabbitMQ, not the cheapest or the most natural for replay. The reason was time to a safe audit trail. The team could put every payout event into Postgres before publishing to RabbitMQ. Postgres already had backups, access controls, and a seven-year retention rule. RabbitMQ then handled delivery, retries, and dead letters. The audit replay came from Postgres, not the broker. That split felt less elegant than Kafka's log, but it used systems the company already operated.
Finance approved $1,100 a month for the managed broker and $6,000 for a part-time contractor to review the Postgres archive. The Kafka quote was $1,000 a month higher and needed a Kafka operator the company did not have. NATS was $200 a month cheaper than RabbitMQ and faster to stand up, but the team could not answer a simple audit question: who on staff can fix a JetStream cluster at 2 a.m. next year? For a payout system, that question carried more weight than a six-minute replay.
The first replay test after cutover found 43 events with missing merchant IDs. The cause was a publisher bug, not the broker. The fix was a database constraint and a second publisher check. The queue now processes 1,900 events a minute at month-end. The audit replay is a SQL query.
More articles