FastAPI Background Tasks for LLM Jobs: A Complete Polling Pattern with Next.js

Your endpoint calls an LLM, the LLM takes eleven seconds to respond, and somewhere between your reverse proxy's timeout and your frontend's fetch call, something gives up waiting. The user sees a spinner, then an error, then a support ticket.

The fix: take the LLM call out of the request path entirely, hand back a job ID immediately, and let the client poll "is it done yet?" until the answer is yes. FastAPI's BackgroundTasks handles the first half; a status endpoint plus client-side polling handles the rest.

The Direct Answer

Run the LLM call inside a FastAPI BackgroundTasks function, generate a job ID before the response goes out, store job state (pending → running → done/failed) somewhere the status endpoint can read it, and return the job ID immediately with a 202. Your Next.js client then polls GET /jobs/{id} every second or two until it sees a terminal status and renders the result.

This works well for a single-process API with modest concurrency and jobs that finish in under a minute or so. It stops working once you need multiple workers, retries, job persistence across restarts, or protection against a slow client hammering your event loop. More on that boundary later — build the simple version first.

FastAPI's BackgroundTasks: What It Actually Does

BackgroundTasks is a small utility built into FastAPI that lets you attach a callable to a response. FastAPI runs that callable after the response has been sent to the client, in the same process, using the same event loop (for async functions) or a thread pool (for sync functions). You inject it like any other dependency:

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()

def write_log(message: str):
    with open("log.txt", "a") as f:
        f.write(message + "\n")

@app.post("/ping")
async def ping(background_tasks: BackgroundTasks):
    background_tasks.add_task(write_log, "ping received")
    return {"status": "ok"}

The client gets {"status": "ok"} immediately; the log write happens after. That's the entire mental model — it's a hook for "run this once the response is on its way out," not a queue, not a scheduler, and not something that survives a process restart.

For LLM jobs, the pattern is: generate a job ID, kick off the LLM call as a background task, and give the client that ID to poll with.

Building the Job Store

Before writing the submit endpoint, you need somewhere to record job state. Two options, depending on what you need:

In-memory dict — good for a single-process API, local development, or low-stakes work where losing job state on a restart is acceptable.

from enum import Enum
from typing import Any
from uuid import uuid4

class JobStatus(str, Enum):
    PENDING = "pending"
    RUNNING = "running"
    DONE = "done"
    FAILED = "failed"

jobs: dict[str, dict[str, Any]] = {}

def create_job() -> str:
    job_id = str(uuid4())
    jobs[job_id] = {"status": JobStatus.PENDING, "result": None, "error": None}
    return job_id

PostgreSQL-backed store — the right call once you have more than one worker process, need jobs to survive a deploy or crash, or want an audit trail.

# CREATE TABLE jobs (
#     id UUID PRIMARY KEY,
#     status TEXT NOT NULL,
#     result JSONB,
#     error TEXT,
#     created_at TIMESTAMPTZ DEFAULT now(),
#     updated_at TIMESTAMPTZ DEFAULT now()
# );

async def create_job(db) -> str:
    job_id = str(uuid4())
    await db.execute(
        "INSERT INTO jobs (id, status) VALUES ($1, $2)",
        job_id, JobStatus.PENDING.value,
    )
    return job_id

async def update_job(db, job_id: str, **fields):
    # build SET clause from fields, always bump updated_at
    ...

Swap the dict calls for DB calls and the rest of the pipeline is identical. Start with the dict if you're unsure. A single-process FastAPI app doesn't need a database for this; just know that a restart or second worker will silently drop job state, so treat it as a placeholder you'll graduate out of.

The Submit Endpoint

from fastapi import BackgroundTasks, FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class GenerateRequest(BaseModel):
    prompt: str

class GenerateResponse(BaseModel):
    job_id: str

async def run_llm_job(job_id: str, prompt: str):
    jobs[job_id]["status"] = JobStatus.RUNNING
    try:
        result = await call_llm(prompt)  # your Anthropic/OpenAI call here
        jobs[job_id]["status"] = JobStatus.DONE
        jobs[job_id]["result"] = result
    except Exception as exc:
        jobs[job_id]["status"] = JobStatus.FAILED
        jobs[job_id]["error"] = str(exc)

@app.post("/jobs", response_model=GenerateResponse, status_code=202)
async def submit_job(req: GenerateRequest, background_tasks: BackgroundTasks):
    job_id = create_job()
    background_tasks.add_task(run_llm_job, job_id, req.prompt)
    return GenerateResponse(job_id=job_id)

The 202 Accepted status code is doing real work — it tells the client "I've taken this, it's not done yet." The run_llm_job function catches its own exceptions and writes them into the job record instead of letting them propagate — by the time it runs, the response has already gone out.

The Status/Result Endpoint

from fastapi.responses import JSONResponse

@app.get("/jobs/{job_id}")
async def get_job(job_id: str):
    job = jobs.get(job_id)
    if job is None:
        raise HTTPException(status_code=404, detail="job not found")

    if job["status"] == JobStatus.FAILED:
        return JSONResponse(
            status_code=200,
            content={"status": job["status"], "error": job["error"]},
        )

    return {"status": job["status"], "result": job["result"]}

Keep this endpoint cheap — it's going to get hit repeatedly. A dict lookup or a single indexed row read is fine.

The Next.js Client: Start and Poll

// useJobPolling.ts
import { useState, useCallback, useRef } from "react";

type JobStatus = "pending" | "running" | "done" | "failed";

interface JobResult<T> {
  status: JobStatus;
  result?: T;
  error?: string;
}

export function useJobPolling<T>(apiBase: string) {
  const [state, setState] = useState<JobResult<T> | null>(null);
  const pollRef = useRef<ReturnType<typeof setInterval> | null>(null);

  const start = useCallback(async (prompt: string) => {
    const res = await fetch(`${apiBase}/jobs`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ prompt }),
    });
    const { job_id } = await res.json();
    setState({ status: "pending" });

    pollRef.current = setInterval(async () => {
      const r = await fetch(`${apiBase}/jobs/${job_id}`);
      const data: JobResult<T> = await r.json();
      setState(data);

      if (data.status === "done" || data.status === "failed") {
        clearInterval(pollRef.current!);
      }
    }, 1500);
  }, [apiBase]);

  return { state, start };
}

Using it in a component:

function GeneratePanel() {
  const { state, start } = useJobPolling<{ text: string }>("/api");

  return (
    <div>
      <button onClick={() => start("Write a product description for a lamp.")}>
        Generate
      </button>
      {state?.status === "running" && <p>Generating…</p>}
      {state?.status === "done" && <p>{state.result?.text}</p>}
      {state?.status === "failed" && <p>Error: {state.error}</p>}
    </div>
  );
}

A 1–2 second interval is reasonable: frequent enough to feel responsive, infrequent enough not to hammer your status endpoint.

Failure Handling, Timeouts, and Retries

Background LLM work fails in ways your happy-path code won't catch on the first pass:

• Always catch inside the task. An uncaught exception in a background task dies quietly in your server logs while the job record sits stuck at "running" forever. Wrap the entire task body in a try/except that writes a clear error to the job record.
• Set a hard timeout on the LLM call itself, separate from any HTTP timeout, so a hung provider connection doesn't leave a job spinning indefinitely. Write a FAILED state with a clear error message if the timeout fires.
• Decide on retry policy explicitly. A transient error (rate limit, momentary 5xx) is often worth one or two automatic retries with backoff before marking the job failed. A validation error or content policy rejection isn't — retrying just burns time and money for the same failure.
• Expire stale jobs. Add a simple sweep (a periodic task, or a check at read time) that marks jobs older than a threshold as failed if they're still "pending" — orphaned records from a server restart shouldn't poll forever.
• Give the client a ceiling. Stop polling and show "still working, check back later" after a reasonable number of attempts rather than polling silently forever.

When BackgroundTasks Stops Being Enough

Stick with BackgroundTasks and an in-memory or single-table job store when:

• You're running a single FastAPI process (or losing jobs to a deploy is acceptable).
• Job volume is low and losing in-flight jobs is an occasional inconvenience.
• You don't need retries beyond simple try/except logic.
• Jobs finish in seconds to low minutes, not hours.

Move to a dedicated task queue (Celery, RQ, Arq, or a managed platform) when:

• You're scaling to multiple API instances and need job state and execution shared across all of them.
• You need real retry policies, dead-letter handling, or job prioritization.
• Jobs need to survive a deploy or crash without vanishing.
• You want to separate "running LLM jobs" from "serving HTTP requests" onto different infrastructure, so a burst of long jobs can't starve your API's ability to answer other requests.
• You need scheduling, chaining multiple jobs together, or visibility tooling beyond what you're willing to hand-build.

The pattern in this article — job ID, status store, polling client — doesn't change when you make that move. What changes is what's behind the submit endpoint: instead of background_tasks.add_task(...), you push a message onto a queue and a separate worker process picks it up, writing to the same job-status table your status endpoint already reads from. That's the real value of building it this way from the start: the frontend contract and polling loop don't get rewritten when the backend outgrows BackgroundTasks.

FAQ

Does FastAPI's BackgroundTasks run in a separate process? No. It runs in the same process as your API, after the response is sent — as an async coroutine on the event loop, or in a thread pool for sync functions. It shares your process's memory and CPU, which is exactly why heavy or numerous background jobs can start to compete with your API's ability to handle other requests.

Will a background task survive if the server restarts mid-job?

Not reliably. With BackgroundTasks, the task stops running if the process exits — there's no persistence or queue-based recovery built in. If you need jobs to survive restarts, you need both persisted job state (PostgreSQL, not a dict) and a queue-based worker that can pick the job back up, which is one of the clearest signals it's time to graduate off BackgroundTasks.

How often should the Next.js client poll? There's no fixed correct number — it's a tradeoff between responsiveness and load on your status endpoint. An interval in the 1–2 second range is a reasonable starting default for most LLM jobs; slow it down for longer-running jobs or add exponential backoff if you expect many concurrent pollers.

Should I use WebSockets or Server-Sent Events instead of polling? They can reduce request overhead and give you push-based updates instead of a client asking repeatedly, but they add real complexity — connection management, reconnection logic, infrastructure that supports long-lived connections. Polling is simpler to build, debug, and deploy, and for most LLM job UIs the extra latency of a 1–2 second poll interval isn't user-visible. Reach for a push-based approach once polling overhead genuinely becomes a burden.