Most complete task queue and workflow engine in pure Python: Horsies

Horsies is currently the most complete pure-Python library that delivers all of the following under one roof:

• First-class DAG workflows (including subworkflows, joins, success policies, and pause/resume)
• Type-safe task and workflow contracts by design
• A transactional claim / finalize / fencing model as the task queue itself
• Strong recovery semantics, immutable attempt history, and operational visibility
• Optimised statements with 0–2 ms server latency
• A static validation gate (horsies check) that turns entire classes of foot-guns into CI or boot-time failures — an invalid schedule, for instance, will not boot the scheduler service

It is not "Celery with a nicer API" and it is not a lightweight Postgres job table wrapper.

It is a complete re-imagining of what a background execution system should look like when correctness, composability, and inspectability matter more than maximum fire-and-forget throughput.

Design philosophy

Its design eliminates the need of hand-rolling a weak state machine around light task queues.

You just need to use the core primitives of Horsies, and you will be able to compose any conditional orchestration flow that your business depends on.

Three common misreads should be avoided from the start:

1. Postgres is the enabling substrate, not a cost-saving substitute for Redis.
2. Pre-1.0 is an API-contract statement, not an engine-maturity statement. Breaking API changes may land in minor releases, documented in the changelog.
3. Horsies is not a plain message queue. Absolute highest message streaming belongs to Redis/Kafka-class systems.

Core primitives: tasks

Every task must return a typed TaskResult[T, TaskError]:

@app.task("add_numbers")
def add_numbers(*, a: int, b: int) -> TaskResult[int, TaskError]:
    return TaskResult(ok=a + b)

Errors are values, no raises.

The library provides structured error-code families (Operational, Contract, Retrieval, Outcome) plus user-defined Domain codes.

Exception mappers (global or per-task) and default_unhandled_error_code let teams keep their own vocabulary while still capturing unhandled crashes as structured data instead of silent log lines.

In other libraries, you would need a log monitoring service to catch those raises, whereas silent raises are not possible by design in Horsies.

Retries are explicit via RetryPolicy (fixed or exponential, with jitter). Timeouts (timeout_ms) SIGKILL the child process; the sibling tasks on that worker recover through the normal crash-recovery path.

Deadlines (good_until) are first-class and produce a distinct EXPIRED outcome.

Pydantic / dataclass support and serialization

You do not have to manually model-validate on the receiving side the way many other task queues force you to.

You can pass Pydantic models and dataclasses like regular Python arguments and return values. The library handles encoding and decoding against the declared types.

from horsies import Horsies, TaskResult, TaskError
from pydantic import BaseModel

app = Horsies(config)

class Order(BaseModel):
    id: int
    items: list[str]

@app.task("process_order")
def process_order(*, order: Order) -> TaskResult[Order, TaskError]:
    return TaskResult(ok=order)

Read more about serialisation in the docs.

Workflows

Workflows are static DAGs of TaskNodes and SubWorkflowNodes.

Two construction styles are supported:

• Functional / dynamic (app.workflow() + .node())
• Class-based (WorkflowDefinition with attribute nodes and Meta)

Nodes support:

• waits_for, args_from / from_node (typed result injection)
• Join modes: all, any, quorum
• allow_failed_deps
• Success policies with required / optional cases
• OnError.FAIL or OnError.PAUSE (human-in-the-loop)
• Subworkflows that become first-class child runs with their own graphs

The engine treats workflow state transitions with the same transactional approach as task claims.

A simple example:

class DeliveryWorkflow(WorkflowDefinition[str]):
    name = "delivery"
    definition_key = "myapp.delivery.v1"

    pickup = TaskNode(fn=pickup_package)
    deliver_door = TaskNode(fn=deliver_to_door, waits_for=[pickup])
    deliver_neighbor = TaskNode(fn=deliver_to_neighbor, waits_for=[pickup])
    deliver_locker = TaskNode(fn=deliver_to_locker, waits_for=[pickup])

    class Meta:
        output = deliver_door
        # Workflow succeeds if ANY delivery method works
        success_policy = SuccessPolicy(cases=[
            SuccessCase(required=[deliver_door]),
            SuccessCase(required=[deliver_neighbor]),
            SuccessCase(required=[deliver_locker]),
        ])

Static validation: horsies check

This is one of the library's most distinctive features.

horsies check myapp.config:app [--live]

It runs a multi-phase validation:

1. Configuration (Pydantic)
2. Task module imports
3. Full DAG validation (cycles, missing deps, type mismatches, serializability, definition keys, and the rest of the HRS-001–HRS-031 family)
4. Execution of @app.workflow_builder functions under send suppression
5. Detection of undecorated builders
6. Policy and schedule safety
7. Optional live broker connectivity

Operational model

• Queues with priority and per-queue concurrency limits
• Cluster-wide caps and soft/hard prefetch controls
• Child-process recycling by task count or RSS memory
• Heartbeats, stale-task reaping, orphaned workflow task termination
• Typed monitoring query API + optional web dashboard (horsies[web]) and TUI
• Schedules: Interval, Hourly, Daily, Weekly, Monthly, and a fully typed CronSchedule that can express constructs classic cron strings struggle with

The dashboard, mid-run

All history lives in ordinary Postgres tables. You can inspect it with SQL, back it up with your normal tools, and reason about it without a proprietary event store.

Performance (reference shape)

On separate machines in the same region against an entry-tier managed Postgres:

• ~2.5 M statements / 24 h across the top-20 statements
• No top-20 statement exceeds 2 ms p99
• Claim function: p50 1 ms / p99 2 ms at ~150 k claims/day

These are server-side per-statement latencies.

The numbers are not the ceiling, just reference. You can achieve higher throughput with the same or better performance depending on your Postgres service.

Showcase: Acme Clothing

You can see it live at horsies-showcase.suleymanozkeskin.com.

The repository includes a full demonstration application: a fictional fast-fashion retailer whose orders, payments, stock, and shipments are real rows in a real Postgres database. External systems are simulated.

It exercises 35 tasks, multiple workflows (including dynamic per-order DAGs and subworkflows), 4 priority queues, 31 schedules, intentional process kills, human-in-the-loop pauses, success policies, quorum joins, timeouts that restart process pools, and full recovery.

The showcase exists to make the library's claims observable rather than theoretical.

Landscape comparison

Horsies occupies a relatively unique niche: a pure library that keeps both the queue and the workflow engine on the same transactional Postgres substrate while giving the developer typed, composable primitives and strong static checks.

When to choose Horsies

Good fit when

• You are building a Python + Postgres product with real multi-step, conditional, failure-aware business processes
• You want orchestration to be expressed as code rather than hand-rolled state machines
• Type safety, early validation, and inspectable history matter
• You are willing to run a dedicated Postgres instance for the broker

Less ideal when

• You only need simple fire-and-forget or delayed jobs
• You require extreme throughput or streaming fan-out
• You need multi-language workers or the absolute most battle-tested durable-execution platform today (Temporal remains stronger there)
• Your team cannot accept a still-evolving (pre-1.0) public API

Current status

Horsies is pre-1.0. The API may still change in minor releases. The engine itself has received substantial investment in failure-path testing, recovery semantics, and cross-validation. Real production use exists, including an HR platform serving hundreds of companies. Public visibility is still low; engineering quality and production evidence are currently stronger signals than GitHub stars.

Closing

Most Python applications that start with a simple task queue eventually grow non-trivial orchestration. At that point teams face a choice: keep bolting state machines onto a visibility-timeout broker, adopt a heavier durable-execution platform, or use a library that treats transactional ownership and typed workflows as core.

Horsies is an opinionated answer to that third path. It is not the right tool for every workload, but for pure-Python, Postgres-centric systems whose background work is actually business logic, it currently offers one of the most coherent and carefully engineered combinations of queue + workflow + static safety available.

The best way to evaluate it is still the same as any serious infrastructure library: read the failure semantics, run the showcase, put horsies check in CI, and measure it against the orchestration you already know you will need.