Building a Webhook Dead Letter Queue for Payment Events

A dead letter queue is where messages go to die. Or more accurately, where messages go when automatic processing has given up on them. The name comes from postal mail: a letter that cannot be delivered and cannot be returned to the sender goes to the dead letter office.

In the context of Paystack webhooks, a DLQ captures events that your queue worker tried to process multiple times and failed every time. Without a DLQ, these events vanish. The queue system discards them, and you have no record that they existed.

For most applications, a discarded email notification or analytics event is annoying but survivable. For payment events, it is a crisis. A discarded charge.success event means a customer paid and your system never recorded it. A discarded transfer.reversed means a payout was clawed back by the bank and your system still shows it as successful. A discarded dispute.create means a chargeback is ticking toward a deadline and nobody on your team knows about it.

The DLQ catches these events. It holds them. It alerts your team. A human investigates, fixes the underlying issue (bad data, schema mismatch, downstream service outage), and replays the event through the normal processing pipeline.

This article is part of the Paystack webhooks engineering guide.

Events should not go to the DLQ on the first failure. Most failures are transient: a database connection timed out, a downstream service was briefly unavailable, a deployment caused a momentary error. Automatic retries handle these.

A good retry policy for payment webhook processing:

• 3 to 5 retries before giving up and moving to the DLQ.
• Exponential backoff between retries: 1 second, then 4 seconds, then 16 seconds, then 64 seconds. This gives transient issues time to resolve without hammering the failing system.
• Jitter on the backoff to prevent all retries from firing at the same instant (thundering herd).

With BullMQ (Node.js + Redis):

const { Queue, Worker } = require('bullmq');
const IORedis = require('ioredis');

const connection = new IORedis(process.env.REDIS_URL);

const webhookQueue = new Queue('paystack-webhooks', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 1000, // Start at 1 second
    },
    removeOnComplete: { count: 1000 },
    removeOnFail: false, // Keep failed jobs so we can move them to DLQ
  },
});

const worker = new Worker('paystack-webhooks', async (job) => {
  await processWebhookEvent(job.data);
}, {
  connection,
  concurrency: 5,
});

// When a job exhausts all retries, move it to the DLQ
worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    // All retries exhausted. Move to DLQ.
    await moveToDeadLetterQueue(job.data, err.message, job.attemptsMade);
  }
});

The removeOnFail: false setting keeps failed jobs in the queue so you can inspect them. The event listener checks whether the job has exhausted all attempts. If so, it moves the event to the DLQ. Otherwise, BullMQ will retry automatically according to the backoff policy.

For more on queue setup, see queueing webhook work with BullMQ.

For most teams, a database table is the best DLQ implementation. It is queryable, persistent, backed up with your regular database backups, and does not require additional infrastructure.

CREATE TABLE webhook_dead_letter_queue (
  id BIGSERIAL PRIMARY KEY,
  event_type VARCHAR(100) NOT NULL,
  reference VARCHAR(255),
  payload JSONB NOT NULL,
  error_message TEXT NOT NULL,
  error_stack TEXT,
  retry_count INTEGER NOT NULL DEFAULT 0,
  original_received_at TIMESTAMPTZ NOT NULL,
  moved_to_dlq_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  status VARCHAR(20) NOT NULL DEFAULT 'pending',
  -- pending, investigating, reprocessed, resolved_manually, discarded
  assigned_to VARCHAR(255),
  resolution_notes TEXT,
  resolved_at TIMESTAMPTZ
);

CREATE INDEX idx_dlq_status ON webhook_dead_letter_queue(status);
CREATE INDEX idx_dlq_event_type ON webhook_dead_letter_queue(event_type);
CREATE INDEX idx_dlq_moved_at ON webhook_dead_letter_queue(moved_to_dlq_at);

The key fields:

• payload: The complete webhook event payload. You need every field to understand the failure and to reprocess the event.
• error_message and error_stack: Why it failed. Without this, the investigator has to guess.
• retry_count: How many times automatic retries attempted processing. Helps distinguish "failed once then went to DLQ" from "failed five times."
• original_received_at: When your server first received the webhook. This tells you how old the event is and how urgently it needs attention.
• status: Tracks the lifecycle of the DLQ entry. Starts as "pending." An engineer investigating it can mark it as "investigating." Once fixed, it becomes "reprocessed" or "resolved_manually."
• assigned_to: Who is looking at it. Prevents two people from working on the same issue.
• resolution_notes: What was wrong and how it was fixed. Future reference for similar failures.

The insert function:

async function moveToDeadLetterQueue(eventData, errorMessage, retryCount) {
  await db.query(
    'INSERT INTO webhook_dead_letter_queue ' +
    '(event_type, reference, payload, error_message, retry_count, original_received_at) ' +
    'VALUES ($1, $2, $3, $4, $5, $6)',
    [
      eventData.eventType,
      eventData.payload.reference || eventData.payload.id,
      JSON.stringify(eventData),
      errorMessage,
      retryCount,
      eventData.receivedAt,
    ]
  );

  // Alert the team
  await sendDlqAlert(eventData.eventType, errorMessage);
}

Every event that lands in the DLQ should trigger an alert. This is not a metric that can wait for a daily report. A charge.success in the DLQ means a customer paid and is waiting.

async function sendDlqAlert(eventType, errorMessage) {
  const severity = getDlqSeverity(eventType);

  const message = {
    text: '[' + severity + '] Paystack webhook moved to DLQ',
    blocks: [
      {
        type: 'section',
        text: {
          type: 'mrkdwn',
          text: '*Webhook Event Failed All Retries*
' +
            'Event: ' + eventType + '
' +
            'Error: ' + errorMessage + '
' +
            'Severity: ' + severity + '
' +
            'Time: ' + new Date().toISOString(),
        },
      },
    ],
  };

  await fetch(process.env.SLACK_WEBHOOK_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(message),
  });

  // For critical events, also page on-call
  if (severity === 'CRITICAL') {
    await pageOnCall(eventType, errorMessage);
  }
}

function getDlqSeverity(eventType) {
  const critical = [
    'charge.success',
    'transfer.success',
    'transfer.failed',
    'transfer.reversed',
    'dispute.create',
  ];

  const high = [
    'refund.processed',
    'refund.failed',
    'subscription.create',
    'subscription.disable',
  ];

  if (critical.includes(eventType)) return 'CRITICAL';
  if (high.includes(eventType)) return 'HIGH';
  return 'MEDIUM';
}

The severity classification matters. A charge.success in the DLQ is a customer not being credited. That is critical and needs immediate attention. A customeridentification.success in the DLQ is important but not urgent. The customer is not losing money over it.

For a comprehensive monitoring setup, see monitoring and alerting on webhook failures.

When an alert fires for a DLQ event, here is the investigation process:

Step 1: Read the error.

SELECT id, event_type, reference, error_message, retry_count, original_received_at
FROM webhook_dead_letter_queue
WHERE status = 'pending'
ORDER BY moved_to_dlq_at DESC;

Step 2: Claim the event.

UPDATE webhook_dead_letter_queue
SET status = 'investigating', assigned_to = 'alice@yourcompany.com'
WHERE id = 42;

Step 3: Diagnose the failure. Common causes:

• Database schema mismatch: A column was renamed, a NOT NULL constraint was added, or a migration was not applied.
• Downstream service failure: The handler called an external API that was down during all retry attempts.
• Data validation error: The payload had an unexpected format (null where a string was expected, a string where a number was expected).
• Business logic bug: The handler has a code path that throws for a specific combination of values.
• Resource exhaustion: The database connection pool was exhausted during a traffic spike.

Step 4: Fix the underlying issue. Deploy the fix (schema migration, code change, config update).

Step 5: Reprocess or resolve manually. Either replay the event through your normal pipeline (next section) or resolve it manually and note what you did.

-- After successful reprocessing
UPDATE webhook_dead_letter_queue
SET status = 'reprocessed',
    resolution_notes = 'Fixed migration. Reprocessed via admin script.',
    resolved_at = NOW()
WHERE id = 42;

-- If resolved manually without reprocessing
UPDATE webhook_dead_letter_queue
SET status = 'resolved_manually',
    resolution_notes = 'Customer already refunded via Paystack dashboard. Updated order manually.',
    resolved_at = NOW()
WHERE id = 42;

Reprocessing a DLQ event means feeding it back through your normal webhook processing pipeline. Because your handler is idempotent (see idempotent webhook handlers), reprocessing is safe even if some partial processing occurred during the original attempts.

// scripts/reprocess-dlq-event.js
// Usage: node reprocess-dlq-event.js --id 42

const { Pool } = require('pg');
const { processEvent } = require('../src/webhook-processor');

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function reprocessDlqEvent(dlqId) {
  const client = await pool.connect();

  try {
    // Fetch the DLQ entry
    const result = await client.query(
      'SELECT * FROM webhook_dead_letter_queue WHERE id = $1',
      [dlqId]
    );

    if (result.rows.length === 0) {
      console.error('DLQ entry not found: ' + dlqId);
      return;
    }

    const dlqEntry = result.rows[0];

    if (dlqEntry.status === 'reprocessed') {
      console.error('DLQ entry already reprocessed: ' + dlqId);
      return;
    }

    console.log(
      'Reprocessing DLQ entry ' + dlqId +
      ': ' + dlqEntry.event_type +
      ' for reference ' + dlqEntry.reference
    );

    // Mark as investigating
    await client.query(
      'UPDATE webhook_dead_letter_queue SET status = $1 WHERE id = $2',
      ['investigating', dlqId]
    );

    // Reprocess through the normal pipeline
    const eventPayload = dlqEntry.payload;
    await processEvent(eventPayload);

    // Mark as reprocessed
    await client.query(
      'UPDATE webhook_dead_letter_queue SET status = $1, ' +
      'resolution_notes = $2, resolved_at = NOW() WHERE id = $3',
      ['reprocessed', 'Reprocessed via admin script', dlqId]
    );

    console.log('Successfully reprocessed DLQ entry ' + dlqId);
  } catch (err) {
    console.error('Reprocessing failed for DLQ entry ' + dlqId + ': ' + err.message);

    // Update the DLQ entry with the new error
    await client.query(
      'UPDATE webhook_dead_letter_queue SET status = $1, ' +
      'error_message = error_message || $2 WHERE id = $3',
      ['pending', '
[Reprocessing attempt] ' + err.message, dlqId]
    );
  } finally {
    client.release();
  }
}

// Bulk reprocessing
async function reprocessAllPending() {
  const result = await pool.query(
    'SELECT id FROM webhook_dead_letter_queue WHERE status = $1 ORDER BY moved_to_dlq_at',
    ['pending']
  );

  console.log('Found ' + result.rows.length + ' pending DLQ entries');

  for (const row of result.rows) {
    await reprocessDlqEvent(row.id);
    // Small delay between events to avoid overwhelming the system
    await new Promise(resolve => setTimeout(resolve, 500));
  }
}

const dlqId = parseInt(process.argv.find(function(arg) { return arg.startsWith('--id='); }).split('=')[1], 10);
reprocessDlqEvent(dlqId).then(function() { process.exit(0); });

Key safety measures:

• Check the DLQ status before reprocessing. Do not reprocess something already reprocessed.
• Use the same processEvent function as the normal webhook pipeline. The idempotency logic protects against double-processing.
• If reprocessing fails again, update the DLQ entry with the new error. Do not delete it.
• For bulk reprocessing, add a delay between events. A burst of reprocessed events can overwhelm your database or downstream services.

For more on safe event replay, see replaying failed webhooks safely.

A DLQ that grows forever becomes a noise factory. Old entries that nobody will investigate should be archived or purged.

-- Archive resolved entries older than 90 days
INSERT INTO webhook_dlq_archive
SELECT * FROM webhook_dead_letter_queue
WHERE resolved_at < NOW() - INTERVAL '90 days';

DELETE FROM webhook_dead_letter_queue
WHERE resolved_at < NOW() - INTERVAL '90 days';

-- Flag stale pending entries (pending for more than 7 days)
UPDATE webhook_dead_letter_queue
SET status = 'stale'
WHERE status = 'pending'
AND moved_to_dlq_at < NOW() - INTERVAL '7 days';

Run these cleanup queries weekly via a scheduled job. Also review stale entries: if something has been pending for 7 days, either investigate it or make a conscious decision to discard it. Leaving it in "pending" status forever pollutes your dashboard and desensitizes your team to DLQ alerts.

For compliance and audit requirements around payment data retention, see webhook logging that survives a compliance audit.

A well-maintained DLQ is a sign of a mature payment system. It means you take seriously every event that Paystack sends, and you have a process for handling the ones that do not go smoothly. Combined with monitoring and alerting, it forms the backbone of payment reliability.