Parallelising Guardrails for Faster Feedback Loops

In AI coding agents need software engineering guardrails, I made the case for why deterministic guardrails matter. In How to build software engineering guardrails: An example with Bylaw, I showed how to think about a guardrail and add one to a project through a concrete example.

Once we know why we need guardrails and how to build them, the next challenge is keeping them out of the way. As the suite grows, the checks need to remain fast enough to run often and shape the work while it is still in progress.

Feedback loops are one of software engineering's advantages

Feedback loops are one of the most valuable things we have when producing work: make a change, observe the result, correct the change, and repeat. The sooner the result arrives, the sooner we can learn from it. Each pass through the loop improves the work until it reaches the quality bar we want.

Software engineers are fortunate to work with unusually short feedback loops, especially in backend development. A compiler, test suite, static analyser, or security scanner can respond in seconds and give us concrete information about what we just changed.

Compare that with a salesperson who may wait months to learn whether an enterprise deal will close, or a payroll administrator whose incorrect entry may surface only during review, reconciliation, or a later pay run. In both cases, the distance between the action and its consequences makes learning slower and attribution harder.

Software engineering does not escape this problem entirely. A compiler can tell us immediately that an abstraction is valid code; it may take six months and several feature requests to discover that it was the wrong abstraction. Our unusually fast feedback applies mainly to properties we can check mechanically.

We should take every opportunity to preserve that advantage. Short feedback loops keep the change and its intent fresh in our minds. They make defects easier to investigate because there is less distance between introducing a problem and learning about it. Slow feedback forces us to restore context and begin a second debugging session.

Guardrails are part of this loop, but only if they are cheap enough to run frequently. A fast quality gate can run before a commit, before a push, in remote CI, and again before merge. The same checks can shape the code throughout the software development lifecycle instead of judging it only at the end.

The challenge is that every new guardrail also makes the feedback loop slower.

Why parallelism is the right lever

As the number of guardrails grows, running them one after another makes the feedback loop progressively slower:

format → compile → Credo → Sobelow → dependency audit → Dialyzer → tests

Most of those checks do not depend on the result of the check before them. They only need the same prepared project. That makes the sequential pipeline unnecessarily expensive.

I changed the quality gate in a small Elixir project to prepare the project once, then run its independent checks concurrently:

deps.get → compile ─┬─ format
                    ├─ Credo
                    ├─ Sobelow
                    ├─ dependency audit
                    ├─ Dialyzer
                    └─ tests

The work has not disappeared. Dependency resolution and compilation still happen in order because the checks depend on their output. The independent verification checks still do exactly the same work as before. Concurrency changes the wall-clock cost: the parallel phase takes roughly as long as its slowest check rather than the sum of every check.

This is the right lever because it preserves the strength of the gate. Removing checks would also make the command faster, but at the cost of feedback. Parallelising independent checks keeps the same coverage while returning that feedback sooner.

The experiment

I measured both the complete command and the verification phase by itself.

For the complete command, the sequential variant ran:

MIX_ENV=test mix deps.get
MIX_ENV=test mix compile --warnings-as-errors
MIX_ENV=test mix format --check-formatted
MIX_ENV=test mix credo --strict
MIX_ENV=test mix sobelow --exit
MIX_ENV=test mix deps.audit
MIX_ENV=test mix dialyzer --no-compile
MIX_ENV=test mix test --no-compile

The parallel variant ran the real mix qa task described below. It performed the same dependency resolution and compilation, then launched the same six checks concurrently.

After one warm-up, I ran five measured trials of each variant. I alternated which variant ran first, used the same warm deps, _build, and Dialyzer PLT, discarded command output, and required every command to exit successfully.

Despite the project's small size, parallelising the verification phase reduced its median duration from 9.676 seconds to 4.300 seconds: a 55.6% reduction. Larger projects will not necessarily see the same proportional improvement, especially when one check, often the test suite, dominates the runtime. Even then, running the remaining checks behind that bottleneck can shave seconds or minutes from every feedback loop.

The isolated verification phase shows the effect of parallelisation most clearly: its median was 2.25 times faster and its parallel results stayed within a 712ms range.

The complete gate includes shared preparation that remains sequential. It also showed more variance. Two parallel trials took 16.719 and 21.739 seconds, while the other three took between 10.263 and 11.434 seconds. deps.get performs dependency resolution before the checks begin, so slow preparation affects either implementation without changing the work that was parallelised.

The project had 352 ExUnit tests. I ran the benchmark on an Apple M4 Max with 14 logical CPUs, macOS 26.5.2, Erlang/OTP 29, and Elixir 1.20.2. These are local development measurements from one machine. The speedup in another project will depend on the duration of its checks, their resource usage, and how much work must remain sequential.

Saving roughly four seconds on a command that runs dozens or hundreds of times per day compounds surprisingly quickly.

Separate preparation from verification

The first decision is which tasks may safely run in parallel.

Dependency resolution and compilation prepare shared state in deps and _build. They run once, in order, before any checks begin:

run_command!("dependencies", ["deps.get"])
run_command!("compile", ["compile", "--warnings-as-errors"])

If either command fails, the task stops. There is little value in launching checks against a project that could not fetch its dependencies or compile successfully.

The remaining checks only inspect the prepared project:

@checks [
  {"format", ["format", "--check-formatted"]},
  {"credo", ["credo", "--strict"]},
  {"sobelow", ["sobelow", "--exit"]},
  {"dependency audit", ["deps.audit"]},
  {"dialyzer", ["dialyzer", "--no-compile"]},
  {"tests", ["test", "--no-compile"]}
]

The --no-compile flags matter. The parent task has already compiled the project with warnings treated as errors. Asking Dialyzer and the test suite to compile again would duplicate work and allow several processes to compete over the same build artifacts.

This gives the quality gate two explicit phases:

1. Prepare shared state sequentially.
2. Verify that state concurrently.

That boundary is the main design constraint. Parallelising commands that mutate the same files or database can introduce races. Independent read-only checks are the safest place to begin.

Run every check in its own Mix process

Mix tasks are designed to run once within a Mix process. They may also change application state, start supervision trees, configure logging, or halt the VM. Running several of them as functions inside one process would couple checks that should be isolated.

Instead, the QA task launches a fresh mix command for every check:

@mix_runner_prefix ["--erl", "-elixir ansi_enabled true", "-S", "mix"]

defp run_command(args) do
  System.cmd(elixir_executable(), @mix_runner_prefix ++ args,
    cd: File.cwd!(),
    env: [{"MIX_ENV", "test"}],
    stderr_to_stdout: true
  )
end

Each command receives the same working directory and MIX_ENV, while keeping its own BEAM VM and Mix task lifecycle. System.cmd/3 returns both the command's output and exit code, which is enough to treat every tool uniformly.

The parent Mix task uses Task.async_stream/3 to run those commands concurrently:

results =
  @checks
  |> Task.async_stream(&run_check/1,
    max_concurrency: Enum.count(@checks),
    ordered: false,
    timeout: :infinity
  )
  |> Enum.map(fn {:ok, result} -> result end)

max_concurrency allows every configured check to start without an artificial queue. ordered: false lets the stream receive completed work immediately. timeout: :infinity leaves timeout policy with the underlying tools, which is useful for test suites and initial Dialyzer runs whose duration varies substantially between machines.

Starting every check at once is appropriate for this project and this number of tasks. A larger suite may need a lower concurrency limit to avoid CPU or memory contention. The useful target is the shortest reliable feedback loop, not the highest possible process count.

Buffer output per check

Concurrent processes create a usability problem: if every command writes directly to the terminal, their output interleaves.

A test failure can appear between two Credo warnings. A Dialyzer message can be split by Sobelow output. The suite may finish sooner while taking longer to understand.

System.cmd/3 buffers each command's output. The QA task prints that buffer only after the command has finished:

defp run_check({label, args}) do
  {output, exit_code} = run_command(args)
  {label, output, exit_code}
end

defp print_result({label, output, exit_code}) do
  status = if exit_code == 0, do: "ok", else: "failed"
  Mix.shell().info("==> #{label} (#{status})")

  if output != "" do
    Mix.shell().info(String.trim_trailing(output))
  end
end

The checks still execute concurrently, but each one appears as a coherent section:

==> format (ok)
==> credo (ok)
...
==> tests (failed)
<complete test output>

This trades live, line-by-line progress for readable results. That is a reasonable default for finite local checks. A command that can produce very large output may need a temporary file or a bounded capture strategy instead of keeping its entire log in memory.

Report all failures

Failing fast is useful during preparation because later checks depend on that state. It is less useful once independent checks are already running.

If formatting, Credo, and tests all fail, I want one run to tell me about all three:

Enum.each(results, &print_result/1)

failures = Enum.reject(results, &match?({_, _, 0}, &1))

if Enum.any?(failures) do
  failed_labels =
    Enum.map_join(failures, ", ", fn {label, _, _} -> label end)

  Mix.raise("QA checks failed: #{failed_labels}")
end

The task waits for every check, prints every result, then exits unsuccessfully with a concise summary. An engineer or coding agent can fix the complete set of known violations before running the gate again.

This is another feedback-loop improvement. Parallel execution reduces the time spent waiting for one run. Aggregating failures reduces the number of runs needed to discover the current state of the branch.

Put the complete gate behind one command

The custom task is exposed through a Mix alias:

defp aliases do
  [
    # ...
    qa: ["qa.run"]
  ]
end

The public command remains:

mix qa

The implementation task is named qa.run because defining Mix.Tasks.Qa while also aliasing qa can create an ambiguous entry point. The alias owns the stable interface; the task owns the orchestration.

One command matters because a guardrail only protects the codebase when it runs consistently. The same complete gate can be called by developers, coding agents, Git hooks, worktree tooling, and CI without each caller reconstructing the list of checks.

The complete task

The excerpts above cover the design decisions. The complete Mix.Tasks.Qa.Run implementation is available as a GitHub Gist.

Faster guardrails get used more often

Adding deterministic guardrails increases confidence, but each new check also adds latency. If that latency grows linearly, a comprehensive quality gate eventually becomes something people postpone until the end of a task.

Independent checks do not need to impose a linear cost. Prepare their shared inputs once, isolate each check in its own process, run them concurrently, keep their output readable, and report all violations together.

The goal is not merely to make CI finish sooner. It is to make the complete set of guardrails cheap enough to use throughout development. A formatting error, unsafe code path, type mismatch, or failing test can then be corrected while the implementation is still fresh in the engineer's or coding agent's context.

Guardrails are most useful when they participate in the edit loop: change the code, run the checks, act on the feedback, and repeat. Making that loop fast is part of designing the guardrails themselves.