The Fan-In That Never Fires

Fan-out is the easy half. Dispatch N tasks, let workers pick them up, walk away. Every workflow engine gets this right.

Fan-in is where it breaks. When those N tasks finish, something has to notice the last one is done and dispatch whatever comes next. The obvious way to do that is correct in a single-threaded test and wrong the moment two workers finish together. It doesn't crash and it doesn't retry. The workflow stops.

The Obvious Fan-In Check

Each worker, on finishing its task, does two things: marks itself complete, then asks whether the whole group is done.

// Mark this task done
task.status = TaskStatus.COMPLETED
taskRepository.save(task)
 
// Is the whole parallel group finished?
val tasks = taskRepository.findByExecutionIdAndParallelGroup(executionId, group)
if (tasks.all { it.status.isTerminal() }) {
    dispatchNext(executionId, group)
}

There's no problem here as long as one task finishes at a time. Every test that completes tasks sequentially passes. So does every test with a parallel block of one.

Why Nobody Advances

PostgreSQL defaults to READ COMMITTED, and so does almost every framework sitting on top of it. Under READ COMMITTED, each statement sees only data committed before that statement began. Your own uncommitted writes are visible to you. Nobody else's are.

Now run two workers finishing at the same instant, each in its own transaction:

At t3, A can't see B's write, because B hasn't committed. At t4, B can't see A's write for the same reason. Both conclude the group is unfinished. Both return. Both commit.

Every task in the group is now COMPLETED in the database, and nothing will look at that group again. The check runs on task completion, and there are no more completions coming.

This is worse than a crash. No exception, no dead letter, no retry, no timeout. Your monitoring shows a workflow in RUNNING with every one of its tasks green. It sits there until someone notices it's been three days.

Why It Doesn't Double-Fire Instead

You'd expect the opposite to be possible too: with different timing, both workers see a complete group and both dispatch downstream. Worth checking. A duplicate downstream task is worse than a stall when that step charges a card.

It can't happen. The transaction boundary rules it out.

Each transaction does three things in a fixed order: write its own status (W), read the siblings (C), then commit. Under READ COMMITTED, C_A observes B as terminal only if B committed first. So for both workers to see a complete group:

C_A sees B  ⟹  commit_B < C_A
C_B sees A  ⟹  commit_A < C_B

but within each transaction:  C_A < commit_A   and   C_B < commit_B

chaining:  commit_B < C_A < commit_A < C_B < commit_B
⟹         commit_B < commit_B          contradiction

At most one worker can observe the group as complete. The argument involves only the last two transactions to commit, so it holds for a group of two or two hundred.

That guarantee comes from keeping the status write and the fan-in check in one transaction. Split them into two and the proof collapses: both workers commit their status, then both check, then both dispatch. So the fix has to preserve that boundary.

Serializing the Decision

Only one worker can run the check at a time, and the loser has to run it after the winner commits. That's a lock. The right row to lock is the parent execution.

@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("SELECT e FROM ExecutionEntity e WHERE e.id = :id")
fun findByIdForUpdate(@Param("id") id: UUID): ExecutionEntity?

Take that lock before counting, inside the same transaction that marked the task complete:

Worker B's read at t5 is a new statement, so it takes a fresh snapshot that includes A's committed write. Zero dispatches becomes one.

Locking the tasks instead wouldn't work. The decision about the group needs a single serialization point, and the execution row is the one thing every task in the group shares.

How Konduit Implements This

Konduit does this in its execution engine, guarded by a comment explaining why the lock is there:

// Acquire pessimistic lock on execution row to serialize fan-in checks.
// Without this lock, concurrent parallel task completions can each see
// incomplete parallel groups (READ COMMITTED isolation), causing none
// to trigger advancement.
val lockedExecution = executionRepository.findByIdForUpdate(executionId)
    ?: throw IllegalStateException("Execution $executionId not found during fan-in check")
 
if (!taskDispatcher.isParallelGroupComplete(executionId, parallelGroup)) {
    return
}
 
val parallelOutputs = taskDispatcher.collectParallelOutputs(executionId, parallelGroup)
val nextTask = taskDispatcher.dispatchNext(/* ... */)

The transaction boundary that makes the proof hold lives one layer up, in a service whose whole job is keeping those two operations together:

@Transactional
fun completeAndAdvance(task: TaskEntity, execution: ExecutionEntity, output: Map<String, Any>?) {
    taskQueue.completeTask(task, output)                // step 1: mark complete
    executionAdvancer.onTaskCompleted(task, execution)  // step 2: fan-in check
}

These used to be separate transactions. A crash landing between them left a task marked COMPLETED and the workflow never advanced, which is the identical symptom from an unrelated cause. Merging them fixed the crash case and, as a side effect, put the double-dispatch interleaving out of reach.

What the Lock Costs

Every sibling completion in a parallel block now serializes on one row, and the lock is held until the transaction commits. That means it's held across dispatchNext, not only across the count. A block of fifty parallel steps finishing together queues fifty transactions behind that row, each waiting on whatever the winner's dispatch does.

I haven't measured this, and I'd rather say so than hand you a number I made up. Fan-in happens once per parallel block rather than once per task execution, so the contended window is short next to the work the tasks themselves are doing. That reasoning is what the design rests on. If you're running very wide blocks, profile it before trusting it.

Two alternatives are worth knowing. An atomic counter, UPDATE ... SET remaining = remaining - 1 RETURNING remaining, makes exactly one caller observe zero without an explicit lock, though it needs its own guard so retries can't decrement twice. A uniqueness constraint on an "already fired" marker row lets everyone race and lets the database pick the winner. Both are good answers. The row lock wins here because the execution row was already being read and written on this path.

Terminal Isn't the Same as Successful

Konduit treats DEAD_LETTER as terminal for fan-in. A parallel step that exhausted its retries stops the group from waiting, but doesn't stop the group from completing.

fun isParallelGroupComplete(executionId: UUID, parallelGroup: String): Boolean {
    val tasks = taskRepository.findByExecutionIdAndParallelGroup(executionId, parallelGroup)
    if (tasks.isEmpty()) return false
    return tasks.all { it.status == TaskStatus.COMPLETED || it.status == TaskStatus.DEAD_LETTER }
}

That's deliberate: failed steps don't cancel their siblings, so work already finished isn't discarded. Output collection then filters to successes, and downstream receives partial results.

The tasks.isEmpty() guard matters more than it looks. Without it, all {} returns true for an empty list, and a group whose tasks haven't been created yet reads as complete.

The Rule

If you're coordinating parallel work in a relational database:

• Never decide "am I the last one?" from a read that isn't serialized against the other finishers
• Always keep the completion write and the completion check in one transaction, which is what rules out double-dispatch
• Always lock the thing the group shares, not the members of the group
• Always test fan-in with tasks finishing simultaneously, because the sequential test passes against broken code

Konduit's parallel tests run against real PostgreSQL 16 in Testcontainers rather than mocks, because this class of bug is invisible to anything without real transactions and real isolation semantics.

A check that reads correctly, passes its tests, and fails only when two things happen at once is the most expensive kind of bug to find. The system never reports it. It stops.

Konduit is a workflow orchestration engine built on PostgreSQL, with SKIP LOCKED task claiming, fan-in coordination, virtual threads, and 184 tests against real PostgreSQL via Testcontainers. See the project page or the source on GitHub.