Juano Morello
Back to projectsCase study

Konduit

Workflow orchestration engine with distributed workers, SKIP LOCKED queues, virtual threads, and fan-in coordination.

View Repository

At a glance

Purpose

Provide a lightweight, self-hosted workflow orchestration engine with proper distributed coordination, task queuing, and failure recovery.

Who it's for

Engineering teams building async job processing systems who need more than a simple queue but less than a full-blown orchestration platform.

Core capabilities
  • Distributed worker coordination with SKIP LOCKED-based task claiming
  • Fan-in coordination for parallel task aggregation
  • Virtual threads for efficient concurrent execution
Stack
KotlinSpring BootPostgreSQLRedisTestcontainers

Context

Many applications need workflow orchestration beyond simple job queues. They need task dependencies, fan-in coordination, failure recovery, and distributed execution. Enterprise solutions exist but are often overkill for mid-scale systems.

The Problem

Simple queues don't handle task dependencies. Enterprise orchestration platforms require significant infrastructure. The middle ground, a lightweight self-hosted orchestration engine, is underserved.

Approach

Konduit uses PostgreSQL's SKIP LOCKED for distributed task claiming, Redis for coordination, and Kotlin coroutines/virtual threads for efficient execution. The architecture is intentionally simple: workers pull tasks, execute them, and report results.

Architecture

Konduit is built around a simple model: workflows contain steps, steps produce tasks, workers execute tasks. PostgreSQL stores workflow state and task queues. Redis handles distributed locking and coordination signals.

The SKIP LOCKED pattern ensures multiple workers can pull tasks from the same queue without conflicts. Each worker claims a batch of tasks atomically, executes them, and marks completion.

Task claiming with SKIP LOCKED
-- Workers claim tasks atomically
SELECT * FROM tasks
WHERE status = 'pending'
  AND scheduled_at <= NOW()
ORDER BY priority DESC, scheduled_at ASC
LIMIT 10
FOR UPDATE SKIP LOCKED;

Distributed Coordination

Fan-in coordination handles the common pattern where multiple parallel tasks must complete before a downstream task can execute. Konduit tracks completion counts and triggers downstream tasks when all dependencies are satisfied.

Virtual threads (Project Loom) enable efficient concurrent execution without the overhead of traditional thread pools. Workers can handle many concurrent tasks without blocking.

Testing Strategy

The test suite uses Testcontainers for realistic integration testing. Real PostgreSQL and Redis instances run in Docker during tests, ensuring behavior matches production. 184 tests cover core functionality, edge cases, and failure scenarios.

Tradeoffs

Konduit optimizes for simplicity over features:

• **PostgreSQL-first**: No pluggable storage backends. PostgreSQL is the source of truth. • **Pull-based workers**: Workers poll for tasks rather than receiving push notifications. • **Single-region**: No built-in multi-region coordination.

These constraints keep the system understandable and operationally simple.