Background Jobs and Task Queues: Celery vs. Arq vs. Postgres-Backed Queues

Every non-trivial backend needs work outside the request/response cycle—send emails, resize images, retry flaky APIs, fan out notifications. None of that belongs in your HTTP handler. The question isn’t whether you need a task queue, but which one you can operate without it becoming a second job.

My honest take after running all three in production.

---

The Contenders

Celery — the industry standard. Python, broker-agnostic (Redis or RabbitMQ), battle-hardened, vast ecosystem. Also: genuinely complex to configure correctly, with a historically painful monitoring story.

Arq — a lightweight async task queue built on Redis and Python’s asyncio. Created by the Pydantic team. Minimal surface area, integrates seamlessly with FastAPI and modern async Python.

Postgres-backed queues — using your existing Postgres database as a queue substrate via SKIP LOCKED or libraries like procrastinate. No additional infrastructure. Transactional guarantees included.

---

When to Use What

Celery: When Scale Demands It

Celery makes sense if:

• You’re running Django (its integration is mature and well-documented)
• You need complex workflows—chains, chords, groups, canvas primitives
• You’re already operating Redis or RabbitMQ
• Multiple teams benefit from established patterns

Critical: don’t use Celery’s default pickle serializer in production. It’s a remote code execution vulnerability. Always set task_serializer = "json" and accept_content = ["json"]:

# celeryconfig.py
broker_url = "redis://localhost:6379/0"
result_backend = "redis://localhost:6379/1"
task_serializer = "json"
result_serializer = "json"
accept_content = ["json"]
task_acks_late = True          # re-queue on worker crash
worker_prefetch_multiplier = 1 # fair task distribution

Most teams miss task_acks_late = True with worker_prefetch_multiplier = 1. Without it, a worker crash silently drops tasks.

Arq: The FastAPI Default

If you’re async-native on FastAPI, Arq is the right choice. It’s minimal, readable, and doesn’t impose separate worker class hierarchies.

# tasks.py
async def send_welcome_email(ctx, user_id: int):
    db = ctx["db"]
    user = await db.get(User, user_id)
    await email_client.send(user.email, template="welcome")

class WorkerSettings:
    functions = [send_welcome_email]
    redis_settings = RedisSettings(host="localhost", port=6379)

Enqueue from FastAPI:

from arq import create_pool

redis = await create_pool(RedisSettings())
await redis.enqueue_job("send_welcome_email", user_id=42)

Arq’s limitations are real: no complex workflow primitives, no built-in periodic scheduling beyond basic cron, smaller community. If you need chains or fan-out patterns, you’ll wire them manually—usually fine for moderate loads. Complexity you don’t import is complexity you don’t debug.

Postgres-Backed Queues: The Underrated Option

Most engineers dismiss this, then rediscover it with relief years later.

If you already run Postgres, adding Redis just for a queue is an infrastructure tax. Postgres with SELECT ... FOR UPDATE SKIP LOCKED gives you a reliable, exactly-once-safe job queue without new moving parts.

Minimal schema:

CREATE TABLE jobs (
  id          BIGSERIAL PRIMARY KEY,
  queue       TEXT NOT NULL DEFAULT 'default',
  payload     JSONB NOT NULL,
  status      TEXT NOT NULL DEFAULT 'pending',
  run_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  locked_at   TIMESTAMPTZ,
  locked_by   TEXT,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_jobs_queue_status_run_at
  ON jobs (queue, status, run_at)
  WHERE status = 'pending';

Worker poll:

UPDATE jobs
SET    status = 'running',
       locked_at = NOW(),
       locked_by = $1
WHERE  id = (
  SELECT id FROM jobs
  WHERE  status = 'pending'
    AND  queue = $2
    AND  run_at <= NOW()
  ORDER BY run_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

SKIP LOCKED is the key. Without it, workers block each other. With it, each worker atomically claims a unique job with no coordination overhead.

The real advantage: transactional enqueueing. You can enqueue a job inside the same transaction that writes its dependencies. If the transaction rolls back, the job vanishes. No orphaned jobs. Redis-backed queues cannot guarantee this without significant complexity.

Use procrastinate if you want a polished wrapper rather than rolling your own.

---

Decision Matrix

Concern	Celery	Arq	Postgres Queue
Existing Postgres only	✗	✗	✓
FastAPI + async-native	✗	✓	✓
Django integration	✓	✗	partial
Complex workflows (chains)	✓	✗	✗
Transactional enqueueing	✗	✗	✓
Operational simplicity	✗	✓	✓
Community ecosystem	✓	✗	✗

---

Final Thoughts

For greenfield FastAPI projects: start with Arq. You’ll know exactly when you’ve outgrown it.

For Django projects: Celery, but configure it correctly from day one. Config debt compounds.

For data integrity–critical work (financial transactions, audit trails, anything verifiable): use Postgres-backed queues. The transactional guarantee outweighs Redis throughput in those contexts.

Don’t mistake Postgres queue simplicity for weakness. Some of the most reliable job systems I’ve operated were just tables and a polling loop.