Product engineering and AI/LLMs 2026-08-13T00:00:00+03:00 https://ryanzidago.com/posts/feed.xml 2026-08-13T00:00:00+03:00 2026-08-13T00:00:00+03:00 https://ryanzidago.com/posts/ai-code-reviewers-should-provide-executable-evidence/ <p>Code-review agents should express actionable findings as failing tests against the expected correct behaviour whenever possible. A code review comment is a claim but a failing test is a claim <strong>and an executable evidence</strong></p> <p>This make findings more reliable, easier to understand, and cheaper to communicate.</p> <p>More reliable:</p> <ul> <li>Running tests verifies the alleged defect exists and filters out false positives.</li> </ul> <p>Easier to understand:</p> <ul> <li>A test demonstrates the problematic behaviour and expected behaviour in executable form.</li> </ul> <p>Cheaper to communicate:</p> <ul> <li>A concise test can replace a lengthy prose explanation, reducing review effort and context window usage (for both humans and AI agents alike!!!) I have concrete examples at work where Claude Opus 5 used 2,892 characters to explain something that Codex had expressed as a failing test in 20 lines of test code.</li> </ul> <h1 id="example">Example</h1> <h2 id="avoid">Avoid</h2> <p>A long prose comment that asks the reviewer to reconstruct and verify the claim:</p> <blockquote> <p><code>Order.total/1</code> appears to allow discounts greater than the subtotal. This can produce a negative total when, for example, an order with a subtotal of 100 cents receives a discount of 150 cents. The total should probably never be lower than zero. Please consider clamping the result or validating the discount before subtracting it.</p> </blockquote> <p>The finding may be correct, but the reviewer still has to determine whether the input is valid, reproduce the issue, and confirm the current behaviour.</p> <h2 id="prefer">Prefer</h2> <p>Express the same finding as a failing test:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-comment">@doc &quot;&quot;&quot;</span></span> <span class="giallo-l"><span class="z-comment">Applying a discount greater than the subtotal produces a negative order total.</span></span> <span class="giallo-l"><span class="z-comment">A negative total could result in an invalid refund or credit instead of a charge.</span></span> <span class="giallo-l"><span class="z-comment">&quot;&quot;&quot;</span></span> <span class="giallo-l"><span>test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">the total cannot be lower than zero</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> order </span><span class="z-keyword">=</span><span> %</span><span class="z-entity z-name">Order</span><span>{</span><span class="z-constant">subtotal</span><span class="z-constant">:</span><span class="z-constant"> 100</span><span>,</span><span class="z-constant"> discount</span><span class="z-constant">:</span><span class="z-constant"> 150</span><span>}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> assert </span><span class="z-entity z-name">Order</span><span>.</span><span class="z-entity z-name">total</span><span>(</span><span>order</span><span>)</span><span class="z-keyword"> ==</span><span class="z-constant"> 0</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The test makes the input, disputed behaviour, and proposed expectation explicit. Running it proves whether the finding applies to the codebase: if the test fails because <code>Order.total/1</code> returns <code>-50</code>, the reviewer has executable evidence of the defect.</p> <p>And if you want to make it even easier for your agents to communicate code review findings, tell them to follow these rules for writing <a rel="noopener nofollow noreferrer external" target="_blank" href="https://ryanzidago.com/posts/self-contained-tests-in-elixir/">self-contained tests in Elixir</a>.</p> <h1 id="two-important-qualifications">Two important qualifications</h1> <h2 id="test-the-expected-behaviour">Test the expected behaviour</h2> <p>The LLM needs to understand your intent unambiguously.</p> <p>I had cases where it was writing the test that proved the error:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">the total is lower than zero</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> org </span><span class="z-keyword">=</span><span> %</span><span class="z-entity z-name">Org</span><span>{</span><span class="z-constant">subtotal</span><span class="z-constant">:</span><span class="z-constant"> 100</span><span>,</span><span class="z-constant"> discout</span><span class="z-constant">:</span><span class="z-constant"> 150</span><span>}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> assert </span><span class="z-entity z-name">Order</span><span>.</span><span class="z-entity z-name">total</span><span>(</span><span>order</span><span>)</span><span class="z-keyword"> ==</span><span class="z-keyword"> -</span><span class="z-constant">50</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>This isn't as useful as declaring what the correct behaviour is and the test will be obsolete as soon as we fix the issue. This is why I use the term "failing test against the expected correct behaviour".</p> <h2 id="not-every-finding-needs-a-test">Not every finding needs a test</h2> <p>When a finding claims that a particular input or state produces incorrect observable behaviour, prefer a failing test. Findings about naming, readability, or architecture may still be better expressed in prose.</p> <p>Some behavioural findings, such as concurrency bugs or unusual edge cases, are also difficult to reproduce reliably. Be cautious about insisting on a failing test in these cases: an AI agent may spend 50 minutes adding substantial test setup and complexity only to produce weak or inconclusive evidence. When the cost of a reliable reproduction is disproportionate, a clear explanation of the risk and the conditions under which it may occur is more useful.</p> 2026-07-25T23:50:01+03:00 2026-07-25T23:50:01+03:00 https://ryanzidago.com/posts/do-we-need-referential-integrity-for-prompts/ <h2 id="problem">Problem</h2> <p>I ran into a problem at work that I suspect will become increasingly common as products allow end users to write and save their own prompts.</p> <p>These prompts rarely exist in isolation. They refer to things in the application: a location, a form field, a role, a workflow step, a report, a policy, or even a particular person.</p> <p>Imagine a hotel group with several properties. Each hotel manager can create a prompt that runs every Monday:</p> <blockquote> <p>Summarize last week’s guest satisfaction scores for the Rosebridge Hotel London and compare them with the previous week.</p> </blockquote> <p>At the time the prompt is written, “Rosebridge Hotel London” is the name of a real location in the system. Six months later, the property is renamed “The Rosebridge Kensington.” The hotel itself has not changed, and it may still have the same internal database ID, but the saved prompt continues to use its old name.</p> <p><strong>Eventually the prompt and the application drift apart.</strong></p> <p>The difficult part is that this does not necessarily produce a clean failure. The prompt is still valid text, so the model will try to interpret it. It may guess that the old and new names refer to the same hotel, fail to find any matching location, or confidently use data from the wrong one. Each outcome wastes tokens and time, but more importantly, it gives the hotel manager an unreliable experience without making the underlying problem obvious.</p> <p>Of course, someone can update the prompt. That is manageable when there is one prompt and everyone remembers it exists. It becomes much harder when customers have created hundreds of prompts across an organization. The application may not know which prompts depend on the entity that changed, so nobody learns about the breakage until users start seeing bad results.</p> <p>Databases have a name for the property that prevents this class of problem: referential integrity. If one record points to another, the database can preserve that relationship or reject a change that would break it. A plain-text prompt has no such protection. It can continue referring to a field, role, or workflow that no longer exists.</p> <p>That led me to this question:</p> <blockquote> <p>If prompts can refer to real entities in a product, do they need something like referential integrity?</p> </blockquote> <p>I explored a few possible approaches, but never arrived at a solution that I found both reliable over time and capable of preserving the simplicity and flexibility of natural-language prompts.</p> 2026-07-24T09:30:00+03:00 2026-07-24T09:30:00+03:00 https://ryanzidago.com/posts/parallelising-guardrails-for-faster-feedback-loops/ <p>In <a href="/posts/ai-coding-agents-need-software-engineering-guardrails/">AI coding agents need software engineering guardrails</a>, I made the case for why deterministic guardrails matter. In <a href="/posts/bylaw-a-concrete-example-of-software-engineering-guardrails/">How to build software engineering guardrails: An example with Bylaw</a>, I showed how to think about a guardrail and add one to a project through a concrete example.</p> <p>Once we know why we need guardrails and how to build them, the next challenge is keeping them out of the way. As the suite grows, the checks need to remain fast enough to run often and shape the work while it is still in progress.</p> <h1 id="feedback-loops-are-one-of-software-engineering-s-advantages">Feedback loops are one of software engineering's advantages</h1> <p><strong>Feedback loops are one of the most valuable things we have</strong> when producing work: make a change, observe the result, correct the change, and repeat. The sooner the result arrives, the sooner we can learn from it. Each pass through the loop improves the work until it reaches the quality bar we want.</p> <p>Software engineers are fortunate to work with unusually short feedback loops, especially in backend development. A compiler, test suite, static analyser, or security scanner can respond in seconds and give us concrete information about what we just changed.</p> <p>Compare that with a salesperson who may wait months to learn whether an enterprise deal will close, or a payroll administrator whose incorrect entry may surface only during review, reconciliation, or a later pay run. In both cases, the distance between the action and its consequences makes learning slower and attribution harder.</p> <p>Software engineering does not escape this problem entirely. A compiler can tell us immediately that an abstraction is valid code; it may take six months and several feature requests to discover that it was the wrong abstraction. Our unusually fast feedback applies mainly to properties we can check mechanically.</p> <p><strong>We should take every opportunity to preserve that advantage.</strong> Short feedback loops keep the change and its intent fresh in our minds. They make defects easier to investigate because there is less distance between introducing a problem and learning about it. Slow feedback forces us to restore context and begin a second debugging session.</p> <p>Guardrails are part of this loop, but only if they are cheap enough to run frequently. A fast quality gate can run before a commit, before a push, in remote CI, and again before merge. The same checks can shape the code throughout the software development lifecycle instead of judging it only at the end.</p> <p>The challenge is that every new guardrail also makes the feedback loop slower.</p> <h1 id="why-parallelism-is-the-right-lever">Why parallelism is the right lever</h1> <p>As the number of guardrails grows, running them one after another makes the feedback loop progressively slower:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>format → compile → Credo → Sobelow → dependency audit → Dialyzer → tests</span></span></code></pre> <p>Most of those checks do not depend on the result of the check before them. They only need the same prepared project. That makes the sequential pipeline unnecessarily expensive.</p> <p>I changed the quality gate in a small Elixir project to prepare the project once, then run its independent checks concurrently:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>deps.get → compile ─┬─ format</span></span> <span class="giallo-l"><span> ├─ Credo</span></span> <span class="giallo-l"><span> ├─ Sobelow</span></span> <span class="giallo-l"><span> ├─ dependency audit</span></span> <span class="giallo-l"><span> ├─ Dialyzer</span></span> <span class="giallo-l"><span> └─ tests</span></span></code></pre> <p>The work has not disappeared. Dependency resolution and compilation still happen in order because the checks depend on their output. The independent verification checks still do exactly the same work as before. Concurrency changes the wall-clock cost: the parallel phase takes roughly as long as its slowest check rather than the sum of every check.</p> <p>This is the right lever because it preserves the strength of the gate. Removing checks would also make the command faster, but at the cost of feedback. Parallelising independent checks keeps the same coverage while returning that feedback sooner.</p> <h1 id="the-experiment">The experiment</h1> <p>I measured both the complete command and the verification phase by itself.</p> <p>For the complete command, the sequential variant ran:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> deps.get</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> compile</span><span class="z-constant"> -</span><span class="z-constant">-warnings-as-errors</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> format</span><span class="z-constant"> -</span><span class="z-constant">-check-formatted</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> credo</span><span class="z-constant"> -</span><span class="z-constant">-strict</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> sobelow</span><span class="z-constant"> -</span><span class="z-constant">-exit</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> deps.audit</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> dialyzer</span><span class="z-constant"> -</span><span class="z-constant">-no-compile</span></span> <span class="giallo-l"><span class="z-variable z-other">MIX_ENV</span><span class="z-keyword">=</span><span class="z-string">t</span><span class="z-string">e</span><span class="z-string">s</span><span class="z-string">t</span><span class="z-entity z-name"> mix</span><span class="z-string"> test</span><span class="z-constant"> -</span><span class="z-constant">-no-compile</span></span></code></pre> <p>The parallel variant ran the real <code>mix qa</code> task described below. It performed the same dependency resolution and compilation, then launched the same six checks concurrently.</p> <p>After one warm-up, I ran five measured trials of each variant. I alternated which variant ran first, used the same warm <code>deps</code>, <code>_build</code>, and Dialyzer PLT, discarded command output, and required every command to exit successfully.</p> <table><thead><tr><th>Scope</th><th style="text-align: right">Sequential median</th><th style="text-align: right">Parallel median</th><th style="text-align: right">Reduction</th><th style="text-align: right">Sequential range</th><th style="text-align: right">Parallel range</th></tr></thead><tbody> <tr><td>Verification checks only</td><td style="text-align: right">9.676s</td><td style="text-align: right">4.300s</td><td style="text-align: right">55.6%</td><td style="text-align: right">9.260–13.030s</td><td style="text-align: right">4.208–4.920s</td></tr> <tr><td>Complete quality gate</td><td style="text-align: right">15.128s</td><td style="text-align: right">11.434s</td><td style="text-align: right">24.4%</td><td style="text-align: right">14.767–18.736s</td><td style="text-align: right">10.263–21.739s</td></tr> </tbody></table> <p>Despite the project's small size, parallelising the verification phase reduced its median duration from 9.676 seconds to 4.300 seconds: a 55.6% reduction. Larger projects will not necessarily see the same proportional improvement, especially when one check, often the test suite, dominates the runtime. Even then, running the remaining checks behind that bottleneck can shave seconds or minutes from every feedback loop.</p> <p>The isolated verification phase shows the effect of parallelisation most clearly: its median was 2.25 times faster and its parallel results stayed within a 712ms range.</p> <p>The complete gate includes shared preparation that remains sequential. It also showed more variance. Two parallel trials took 16.719 and 21.739 seconds, while the other three took between 10.263 and 11.434 seconds. <code>deps.get</code> performs dependency resolution before the checks begin, so slow preparation affects either implementation without changing the work that was parallelised.</p> <p>The project had 352 ExUnit tests. I ran the benchmark on an Apple M4 Max with 14 logical CPUs, macOS 26.5.2, Erlang/OTP 29, and Elixir 1.20.2. These are local development measurements from one machine. The speedup in another project will depend on the duration of its checks, their resource usage, and how much work must remain sequential.</p> <p>Saving roughly four seconds on a command that runs dozens or hundreds of times per day compounds surprisingly quickly.</p> <h1 id="separate-preparation-from-verification">Separate preparation from verification</h1> <p>The first decision is which tasks may safely run in parallel.</p> <p>Dependency resolution and compilation prepare shared state in <code>deps</code> and <code>_build</code>. They run once, in order, before any checks begin:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-entity z-name">run_command!</span><span>(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">dependencies</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">deps.get</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>)</span></span> <span class="giallo-l"><span class="z-entity z-name">run_command!</span><span>(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">compile</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">compile</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">--warnings-as-errors</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>)</span></span></code></pre> <p>If either command fails, the task stops. There is little value in launching checks against a project that could not fetch its dependencies or compile successfully.</p> <p>The remaining checks only inspect the prepared project:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-variable z-other">@</span><span class="z-variable z-other">checks</span><span> [</span></span> <span class="giallo-l"><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">format</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">format</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">--check-formatted</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">credo</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">credo</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">--strict</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">sobelow</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">sobelow</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">--exit</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">dependency audit</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">deps.audit</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">dialyzer</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">dialyzer</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">--no-compile</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">tests</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">test</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">--no-compile</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span><span>}</span></span> <span class="giallo-l"><span>]</span></span></code></pre> <p>The <code>--no-compile</code> flags matter. The parent task has already compiled the project with warnings treated as errors. Asking Dialyzer and the test suite to compile again would duplicate work and allow several processes to compete over the same build artifacts.</p> <p>This gives the quality gate two explicit phases:</p> <ol> <li>Prepare shared state sequentially.</li> <li>Verify that state concurrently.</li> </ol> <p>That boundary is the main design constraint. Parallelising commands that mutate the same files or database can introduce races. Independent read-only checks are the safest place to begin.</p> <h1 id="run-every-check-in-its-own-mix-process">Run every check in its own Mix process</h1> <p>Mix tasks are designed to run once within a Mix process. They may also change application state, start supervision trees, configure logging, or halt the VM. Running several of them as functions inside one process would couple checks that should be isolated.</p> <p>Instead, the QA task launches a fresh <code>mix</code> command for every check:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-variable z-other">@</span><span class="z-variable z-other">mix_runner_prefix</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">--erl</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">-elixir ansi_enabled true</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">-S</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">mix</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">defp</span><span class="z-entity z-name"> run_command</span><span>(</span><span>args</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-entity z-name"> System</span><span>.</span><span class="z-entity z-name">cmd</span><span>(</span><span class="z-entity z-name">elixir_executable</span><span>(</span><span>)</span><span>,</span><span class="z-variable z-other"> @</span><span class="z-variable z-other">mix_runner_prefix</span><span class="z-keyword"> ++</span><span> args</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> cd</span><span class="z-constant">:</span><span class="z-entity z-name"> File</span><span>.</span><span class="z-entity z-name">cwd!</span><span>(</span><span>)</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> env</span><span class="z-constant">:</span><span> [</span><span>{</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">MIX_ENV</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">test</span><span class="z-punctuation z-definition z-string">&quot;</span><span>}</span><span>]</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> stderr_to_stdout</span><span class="z-constant">:</span><span class="z-constant"> true</span></span> <span class="giallo-l"><span> )</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Each command receives the same working directory and <code>MIX_ENV</code>, while keeping its own BEAM VM and Mix task lifecycle. <code>System.cmd/3</code> returns both the command's output and exit code, which is enough to treat every tool uniformly.</p> <p>The parent Mix task uses <code>Task.async_stream/3</code> to run those commands concurrently:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>results </span><span class="z-keyword">=</span></span> <span class="giallo-l"><span class="z-variable z-other"> @</span><span class="z-variable z-other">checks</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Task</span><span>.</span><span class="z-entity z-name">async_stream</span><span>(</span><span class="z-variable z-other">&amp;</span><span>run_check</span><span class="z-keyword">/</span><span class="z-constant">1</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> max_concurrency</span><span class="z-constant">:</span><span class="z-entity z-name"> Enum</span><span>.</span><span class="z-entity z-name">count</span><span>(</span><span class="z-variable z-other">@</span><span class="z-variable z-other">checks</span><span>)</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> ordered</span><span class="z-constant">:</span><span class="z-constant"> false</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> timeout</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">infinity</span></span> <span class="giallo-l"><span> )</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Enum</span><span>.</span><span class="z-entity z-name">map</span><span>(</span><span class="z-keyword">fn</span><span> {</span><span class="z-constant">:</span><span class="z-constant">ok</span><span>,</span><span> result</span><span>}</span><span class="z-keyword"> -&gt;</span><span> result </span><span class="z-keyword">end</span><span>)</span></span></code></pre> <p><code>max_concurrency</code> allows every configured check to start without an artificial queue. <code>ordered: false</code> lets the stream receive completed work immediately. <code>timeout: :infinity</code> leaves timeout policy with the underlying tools, which is useful for test suites and initial Dialyzer runs whose duration varies substantially between machines.</p> <p>Starting every check at once is appropriate for this project and this number of tasks. A larger suite may need a lower concurrency limit to avoid CPU or memory contention. The useful target is the shortest reliable feedback loop, not the highest possible process count.</p> <h1 id="buffer-output-per-check">Buffer output per check</h1> <p>Concurrent processes create a usability problem: if every command writes directly to the terminal, their output interleaves.</p> <p>A test failure can appear between two Credo warnings. A Dialyzer message can be split by Sobelow output. The suite may finish sooner while taking longer to understand.</p> <p><code>System.cmd/3</code> buffers each command's output. The QA task prints that buffer only after the command has finished:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">defp</span><span class="z-entity z-name"> run_check</span><span>(</span><span>{</span><span>label</span><span>,</span><span> args</span><span>}</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> {</span><span>output</span><span>,</span><span> exit_code</span><span>}</span><span class="z-keyword"> =</span><span class="z-entity z-name"> run_command</span><span>(</span><span>args</span><span>)</span></span> <span class="giallo-l"><span> {</span><span>label</span><span>,</span><span> output</span><span>,</span><span> exit_code</span><span>}</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">defp</span><span class="z-entity z-name"> print_result</span><span>(</span><span>{</span><span>label</span><span>,</span><span> output</span><span>,</span><span> exit_code</span><span>}</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> status </span><span class="z-keyword">=</span><span class="z-keyword"> if</span><span> exit_code </span><span class="z-keyword">==</span><span class="z-constant"> 0</span><span>,</span><span class="z-constant"> do</span><span class="z-constant">:</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">ok</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-constant"> else</span><span class="z-constant">:</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">failed</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name"> Mix</span><span>.</span><span class="z-entity z-name">shell</span><span>(</span><span>)</span><span>.</span><span class="z-entity z-name">info</span><span>(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">==&gt; </span><span class="z-string">#{</span><span class="z-string">label</span><span class="z-string">}</span><span class="z-string"> (</span><span class="z-string">#{</span><span class="z-string">status</span><span class="z-string">}</span><span class="z-string">)</span><span class="z-punctuation z-definition z-string">&quot;</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword"> if</span><span> output </span><span class="z-keyword">!=</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-entity z-name"> Mix</span><span>.</span><span class="z-entity z-name">shell</span><span>(</span><span>)</span><span>.</span><span class="z-entity z-name">info</span><span>(</span><span class="z-entity z-name">String</span><span>.</span><span class="z-entity z-name">trim_trailing</span><span>(</span><span>output</span><span>)</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> end</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The checks still execute concurrently, but each one appears as a coherent section:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>==&gt; format (ok)</span></span> <span class="giallo-l"><span>==&gt; credo (ok)</span></span> <span class="giallo-l"><span>...</span></span> <span class="giallo-l"><span>==&gt; tests (failed)</span></span> <span class="giallo-l"><span>&lt;complete test output&gt;</span></span></code></pre> <p>This trades live, line-by-line progress for readable results. That is a reasonable default for finite local checks. A command that can produce very large output may need a temporary file or a bounded capture strategy instead of keeping its entire log in memory.</p> <h1 id="report-all-failures">Report all failures</h1> <p>Failing fast is useful during preparation because later checks depend on that state. It is less useful once independent checks are already running.</p> <p>If formatting, Credo, and tests all fail, I want one run to tell me about all three:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-entity z-name">Enum</span><span>.</span><span class="z-entity z-name">each</span><span>(</span><span>results</span><span>,</span><span class="z-variable z-other"> &amp;</span><span>print_result</span><span class="z-keyword">/</span><span class="z-constant">1</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>failures </span><span class="z-keyword">=</span><span class="z-entity z-name"> Enum</span><span>.</span><span class="z-entity z-name">reject</span><span>(</span><span>results</span><span>,</span><span class="z-variable z-other"> &amp;</span><span class="z-entity z-name">match?</span><span>(</span><span>{</span><span class="z-comment">_</span><span>,</span><span class="z-comment"> _</span><span>,</span><span class="z-constant"> 0</span><span>}</span><span>,</span><span class="z-variable z-other"> &amp;</span><span class="z-variable z-other">1</span><span>)</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">if</span><span class="z-entity z-name"> Enum</span><span>.</span><span class="z-entity z-name">any?</span><span>(</span><span>failures</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> failed_labels </span><span class="z-keyword">=</span></span> <span class="giallo-l"><span class="z-entity z-name"> Enum</span><span>.</span><span class="z-entity z-name">map_join</span><span>(</span><span>failures</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">, </span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-keyword"> fn</span><span> {</span><span>label</span><span>,</span><span class="z-comment"> _</span><span>,</span><span class="z-comment"> _</span><span>}</span><span class="z-keyword"> -&gt;</span><span> label </span><span class="z-keyword">end</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name"> Mix</span><span>.</span><span class="z-entity z-name">raise</span><span>(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">QA checks failed: </span><span class="z-string">#{</span><span class="z-string">failed_labels</span><span class="z-string">}</span><span class="z-punctuation z-definition z-string">&quot;</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The task waits for every check, prints every result, then exits unsuccessfully with a concise summary. An engineer or coding agent can fix the complete set of known violations before running the gate again.</p> <p>This is another feedback-loop improvement. Parallel execution reduces the time spent waiting for one run. Aggregating failures reduces the number of runs needed to discover the current state of the branch.</p> <h1 id="put-the-complete-gate-behind-one-command">Put the complete gate behind one command</h1> <p>The custom task is exposed through a Mix alias:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">defp</span><span class="z-entity z-name"> aliases</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> [</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> #</span><span class="z-comment"> ...</span></span> <span class="giallo-l"><span class="z-constant"> qa</span><span class="z-constant">:</span><span> [</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">qa.run</span><span class="z-punctuation z-definition z-string">&quot;</span><span>]</span></span> <span class="giallo-l"><span> ]</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The public command remains:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">mix</span><span class="z-string"> qa</span></span></code></pre> <p>The implementation task is named <code>qa.run</code> because defining <code>Mix.Tasks.Qa</code> while also aliasing <code>qa</code> can create an ambiguous entry point. The alias owns the stable interface; the task owns the orchestration.</p> <p>One command matters because a guardrail only protects the codebase when it runs consistently. The same complete gate can be called by developers, coding agents, Git hooks, worktree tooling, and CI without each caller reconstructing the list of checks.</p> <h1 id="the-complete-task">The complete task</h1> <p>The excerpts above cover the design decisions. The <a rel="noopener nofollow noreferrer external" target="_blank" href="https://gist.github.com/ryanzidago/3f4151196b20db7c1f0b4ca0d819d42a">complete <code>Mix.Tasks.Qa.Run</code> implementation</a> is available as a GitHub Gist.</p> <h1 id="faster-guardrails-get-used-more-often">Faster guardrails get used more often</h1> <p>Adding deterministic guardrails increases confidence, but each new check also adds latency. If that latency grows linearly, a comprehensive quality gate eventually becomes something people postpone until the end of a task.</p> <p>Independent checks do not need to impose a linear cost. Prepare their shared inputs once, isolate each check in its own process, run them concurrently, keep their output readable, and report all violations together.</p> <p>The goal is not merely to make CI finish sooner. It is to make the complete set of guardrails cheap enough to use throughout development. A formatting error, unsafe code path, type mismatch, or failing test can then be corrected while the implementation is still fresh in the engineer's or coding agent's context.</p> <p>Guardrails are most useful when they participate in the edit loop: change the code, run the checks, act on the feedback, and repeat. Making that loop fast is part of designing the guardrails themselves.</p> 2026-07-23T12:00:00+03:00 2026-07-23T12:00:00+03:00 https://ryanzidago.com/posts/self-contained-tests-in-elixir/ <p>A test should explain itself.</p> <p>Every time I have to jump to a setup callback to understand why an assertion passes, the test has already failed at its second job: communication.</p> <p>I understand why they are appealing. A <a rel="noopener nofollow noreferrer external" target="_blank" href="https://ex-unit.hexdocs.pm/ExUnit.Callbacks.html#setup/1"><code>setup</code> callback</a> removes repetition, makes each test shorter, and gives us one place to create the records that several tests need.</p> <p>But shorter tests are not necessarily simpler tests.</p> <p>Very often, shared setup makes a test harder to read, harder to review, harder to change, and harder to move. It reduces the number of lines in the test by moving essential information somewhere else.</p> <p>I much prefer self-contained tests: each test should create and own the state it needs.</p> <h1 id="a-short-test-with-hidden-context">A short test with hidden context</h1> <p>Here is a fairly typical Elixir test:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">defmodule</span><span class="z-entity"> MyApp</span><span>.</span><span class="z-entity z-name">OrdersTest</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-keyword"> use</span><span class="z-entity z-name"> MyApp</span><span>.</span><span class="z-entity z-name">DataCase</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> setup </span><span class="z-keyword">do</span></span> <span class="giallo-l"><span> customer </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">customer</span><span>)</span></span> <span class="giallo-l"><span> product </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">product</span><span>,</span><span class="z-constant"> price</span><span class="z-constant">:</span><span class="z-constant"> 1_000</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> %</span><span>{</span><span class="z-constant">customer</span><span class="z-constant">:</span><span> customer</span><span>,</span><span class="z-constant"> product</span><span class="z-constant">:</span><span> product</span><span>}</span></span> <span class="giallo-l"><span class="z-keyword"> end</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">uses the product price as the order total</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span> %</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> customer</span><span class="z-constant">:</span><span> customer</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> product</span><span class="z-constant">:</span><span> product</span></span> <span class="giallo-l"><span> }</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> assert </span><span>{</span><span class="z-constant">:</span><span class="z-constant">ok</span><span>,</span><span> order</span><span>}</span><span class="z-keyword"> =</span><span class="z-entity z-name"> Orders</span><span>.</span><span class="z-entity z-name">create</span><span>(</span><span>customer</span><span>,</span><span> product</span><span>)</span></span> <span class="giallo-l"><span> assert order</span><span>.</span><span>total </span><span class="z-keyword">==</span><span class="z-constant"> 1_000</span></span> <span class="giallo-l"><span class="z-keyword"> end</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The test itself is short, but it is not self-explanatory.</p> <p>Why should the total be <code>1_000</code>? The answer is not in the test. We need to leave it, find the relevant <code>setup</code>, notice the product's price, and remember that detail when we return.</p> <p>The setup is not incidental. It contains the condition that causes the behaviour we are testing.</p> <p>I would write the test like this:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">uses the product price as the order total</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> customer </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">customer</span><span>)</span></span> <span class="giallo-l"><span> product </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">product</span><span>,</span><span class="z-constant"> price</span><span class="z-constant">:</span><span class="z-constant"> 1_000</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> assert </span><span>{</span><span class="z-constant">:</span><span class="z-constant">ok</span><span>,</span><span> order</span><span>}</span><span class="z-keyword"> =</span><span class="z-entity z-name"> Orders</span><span>.</span><span class="z-entity z-name">create</span><span>(</span><span>customer</span><span>,</span><span> product</span><span>)</span></span> <span class="giallo-l"><span> assert order</span><span>.</span><span>total </span><span class="z-keyword">==</span><span class="z-constant"> 1_000</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Now the test tells the whole story:</p> <ul> <li>the product costs <code>1_000</code></li> <li>creating the order results in a total of <code>1_000</code></li> </ul> <p>There are no important facts hidden elsewhere in the module.</p> <h1 id="locality-matters-more-than-line-count">Locality matters more than line count</h1> <p>Shared setup is usually introduced to remove duplication. That is a useful goal in production code, but I do not think it should be applied mechanically to tests.</p> <p>Production code and test code have different jobs. Production code primarily implements behaviour. A test is also an explanation and an example.</p> <p>Some repetition helps that explanation.</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">creates an order when the product is in stock</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> customer </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">customer</span><span>)</span></span> <span class="giallo-l"><span> product </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">product</span><span>,</span><span class="z-constant"> stock</span><span class="z-constant">:</span><span class="z-constant"> 1</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> assert </span><span>{</span><span class="z-constant">:</span><span class="z-constant">ok</span><span>,</span><span> order</span><span>}</span><span class="z-keyword"> =</span><span class="z-entity z-name"> Orders</span><span>.</span><span class="z-entity z-name">create</span><span>(</span><span>customer</span><span>,</span><span> product</span><span>)</span></span> <span class="giallo-l"><span> assert order</span><span>.</span><span>product_id </span><span class="z-keyword">==</span><span> product</span><span>.</span><span>id</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">rejects an order when the product is out of stock</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> customer </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">customer</span><span>)</span></span> <span class="giallo-l"><span> product </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">product</span><span>,</span><span class="z-constant"> stock</span><span class="z-constant">:</span><span class="z-constant"> 0</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> assert </span><span>{</span><span class="z-constant">:</span><span class="z-constant">error</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">out_of_stock</span><span>}</span><span class="z-keyword"> =</span></span> <span class="giallo-l"><span class="z-entity z-name"> Orders</span><span>.</span><span class="z-entity z-name">create</span><span>(</span><span>customer</span><span>,</span><span> product</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Those tests repeat several lines, but the repetition makes the difference between the two cases immediately visible.</p> <p>If the customer and product were created in a setup callback, we would save a few lines while making the reader reconstruct each scenario from multiple places.</p> <p>That is not a good trade.</p> <p>The useful question is not:</p> <blockquote> <p>How do I remove every repeated line?</p> </blockquote> <p>It is:</p> <blockquote> <p>How much context does someone need to understand this test?</p> </blockquote> <p>For me, a few duplicated fixture calls are usually cheaper than scattered context.</p> <h1 id="self-contained-tests-make-better-review-comments">Self-contained tests make better review comments</h1> <p>The cost of shared setup becomes particularly visible during code review.</p> <p>A reviewer rarely reads a test file from top to bottom with every detail loaded in their head. They often encounter a test in a pull request diff, an inline comment, a suggested patch, or a general PR conversation.</p> <p>Imagine that an AI agent flags a potential bug while reviewing a pull request. I ask it to draft a failing test that demonstrates the issue. I often find a concrete test case easier to understand than a description in plain English. I run the test locally, confirm that it fails for the expected reason, and paste it into a GitHub comment:</p> <p><img src="/images/self-contained-test-review-comment.svg" alt="A GitHub review comment containing a self-contained failing test" /></p> <p>That comment is useful on its own. The author, another reviewer, or an AI agent can understand the scenario without access to anything else.</p> <p>Now imagine that the same test relies on the module's shared setup:</p> <p><img src="/images/shared-setup-test-review-comment.svg" alt="A GitHub review comment containing a test that depends on hidden shared setup" /></p> <p>What kind of product did the setup create? Was it available before the update? Does the setup create anything else that affects the result?</p> <p>The test may make sense inside the file, after jumping to the setup callback and possibly through one or two helper functions. It does not make sense in isolation.</p> <p>This creates unnecessary work for every reviewer. They must open the file, find the setup, understand which parts apply, and mentally combine that information with the test in the comment.</p> <p>It is even worse for an AI agent. We either need to provide the entire file as context, trust the agent to locate every indirect dependency, or accept that it may reason about an incomplete scenario.</p> <p>Self-contained tests reduce that context requirement. They make the evidence for a review comment portable.</p> <h1 id="tests-should-be-portable">Tests should be portable</h1> <p>I want to be able to copy a test from:</p> <ul> <li>my editor into a pull request comment</li> <li>a pull request comment into my local repository</li> <li>one test module into another</li> <li>an issue into an AI coding agent conversation</li> <li>an AI agent's response into the codebase</li> </ul> <p>The test should carry the facts required to understand it.</p> <p>Of course, no test is literally independent of the application. It will still use factories, application modules, and common test infrastructure. Portability is not about eliminating every dependency. It is about making the dependencies that define the scenario explicit at the point of use.</p> <p>A shared setup callback couples a test to its location in the file. Move the test and it may stop compiling, stop passing, or, more dangerously, continue passing under different assumptions.</p> <p>A self-contained test can usually be moved without bringing along a web of setup callbacks. If it cannot, the missing dependency tends to be obvious.</p> <h1 id="shared-setup-accumulates-stale-state">Shared setup accumulates stale state</h1> <p>Shared setup also has a tendency to grow.</p> <p>A test needs an account, so we add an account. Another test needs an administrator, so we add an administrator. A third needs a subscription, so we add a subscription and perhaps a helper that constructs it.</p> <p>Eventually the setup creates a small world:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>setup </span><span class="z-keyword">do</span></span> <span class="giallo-l"><span> account </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">account</span><span>)</span></span> <span class="giallo-l"><span> admin </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">user</span><span>,</span><span class="z-constant"> role</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">admin</span><span>,</span><span class="z-constant"> account</span><span class="z-constant">:</span><span> account</span><span>)</span></span> <span class="giallo-l"><span> employee </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">user</span><span>,</span><span class="z-constant"> role</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">employee</span><span>,</span><span class="z-constant"> account</span><span class="z-constant">:</span><span> account</span><span>)</span></span> <span class="giallo-l"><span> subscription </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">subscription</span><span>,</span><span class="z-constant"> account</span><span class="z-constant">:</span><span> account</span><span>)</span></span> <span class="giallo-l"><span> product </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">product</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> %</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> account</span><span class="z-constant">:</span><span> account</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> admin</span><span class="z-constant">:</span><span> admin</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> employee</span><span class="z-constant">:</span><span> employee</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> subscription</span><span class="z-constant">:</span><span> subscription</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> product</span><span class="z-constant">:</span><span> product</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Not every test needs that world, but every test pays for it.</p> <p>Later, tests are removed or rewritten, while setup entries and helper functions remain. It becomes difficult to tell whether a fixture is still necessary because its consumers are distributed across the module. Unused values, stale helpers, and accidental dependencies survive much longer than they would if the setup were local.</p> <p>Deep setup abstractions also make intermittent failures harder to investigate. When a test fails only occasionally, the first task is to reconstruct its exact starting state. If that state is spread across setup callbacks and helper functions, it becomes difficult to determine what data was created, which defaults mattered, and whether the test accidentally depended on state it did not explicitly request. Self-contained tests do not prevent every flaky test, but they reduce the amount of hidden context we must inspect when one fails.</p> <p>Local setup makes dead code easier to see. If a test no longer uses <code>subscription</code>, we remove the line directly above it. There is no need to audit every other test before deciding whether it is safe.</p> <p>It also makes changes easier to reason about. Changing a shared setup value may affect an entire module, including tests that do not visibly reference the changed detail. Changing local setup affects the test in front of us.</p> <p>This is the same reason global state is difficult to reason about, only at a smaller scale: the distance between cause and effect makes the impact of a change less obvious.</p> <h1 id="self-contained-does-not-mean-no-abstraction">Self-contained does not mean no abstraction</h1> <p>I am not arguing that every test should manually build every struct or repeat a dozen low-level database operations.</p> <p>Helpers are useful when they express domain intent:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>test </span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">a customer cannot order an unavailable product</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> customer </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">customer</span><span>)</span></span> <span class="giallo-l"><span> product </span><span class="z-keyword">=</span><span class="z-entity z-name"> unavailable_product</span><span>(</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> assert </span><span>{</span><span class="z-constant">:</span><span class="z-constant">error</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">product_unavailable</span><span>}</span><span class="z-keyword"> =</span></span> <span class="giallo-l"><span class="z-entity z-name"> Orders</span><span>.</span><span class="z-entity z-name">create</span><span>(</span><span>customer</span><span>,</span><span> product</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The test still owns its setup. It explicitly asks for the state it needs, and the helper gives that state a meaningful name.</p> <p>The important distinction is control:</p> <ul> <li>a helper runs because the test calls it</li> <li>a setup callback runs because the test happens to live in a module</li> </ul> <p>Explicit helpers can make tests clearer. Implicit setup makes their dependencies easier to miss.</p> <p>I would still be careful with helpers that hide the exact value under test. If product availability is central to the behaviour, spelling it out may be clearer:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>product </span><span class="z-keyword">=</span><span class="z-entity z-name"> insert</span><span>(</span><span class="z-constant">:</span><span class="z-constant">product</span><span>,</span><span class="z-constant"> available</span><span class="z-constant">:</span><span class="z-constant"> false</span><span>)</span></span></code></pre> <p>The goal is not to inline everything. The goal is to keep the facts that explain the expected result close to the assertion.</p> <h1 id="the-economics-of-repetition-have-changed">The economics of repetition have changed</h1> <p>Shared setup was more appealing when every repeated fixture had to be written and maintained manually. Even then, I disliked the tradeoff: we saved a few keystrokes by moving important context away from the test.</p> <p>AI coding agents have weakened that tradeoff even further. Asking an agent to generate a few explicit fixture calls costs almost nothing. The expensive part is no longer producing those lines. It is reading, reviewing, and reasoning about them later.</p> <p>AI has made code cheaper to write, but it has not made hidden context cheaper to understand. If an abstraction saves typing while making every test harder to comprehend, I no longer see much value in it.</p> <h1 id="when-i-still-use-setup">When I still use <code>setup</code></h1> <p>Elixir's <code>setup</code> is not inherently bad. Uniform infrastructure that is not part of the scenario being described is the strongest case for using it.</p> <p>For example, a Phoenix controller test might apply the same request header to every connection:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>setup %</span><span>{</span><span class="z-constant">conn</span><span class="z-constant">:</span><span> conn</span><span>}</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> {</span><span class="z-constant">:</span><span class="z-constant">ok</span><span>,</span><span class="z-constant"> conn</span><span class="z-constant">:</span><span class="z-entity z-name"> put_req_header</span><span>(</span><span>conn</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">accept</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">application/json</span><span class="z-punctuation z-definition z-string">&quot;</span><span>)</span><span>}</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>That configuration is different from silently deciding that every test has a customer, an active subscription, and an available product worth <code>1_000</code>.</p> <p>Even in cases like this, I generally prefer to avoid <code>setup</code>. I would rather keep the convention unambiguous: each test owns the state it needs.</p> <p>Once a setup callback exists, it becomes the obvious place to put the next shared dependency. A human sees the existing pattern and extends it. An AI agent does the same. The callback that started with harmless infrastructure gradually accumulates domain state, and the boundary between the two becomes less clear.</p> <p>Repeating a small amount of infrastructure is often a reasonable price for making the preferred pattern obvious to every contributor, human or agent.</p> <h1 id="each-test-should-own-its-world">Each test should own its world</h1> <p>Self-contained tests are sometimes longer. I am happy to accept that.</p> <p>What bothers me is not abstraction itself. A domain function or module can also hide implementation details, but it is part of the product and carries domain meaning. Calling it from a test still tells the reader something about the system.</p> <p>A setup callback introduces a different kind of abstraction: an ad hoc API that exists only between the setup and its tests. That API is implicit, specific to the test module, and has no meaning outside it. I do not want readers to learn both the product's API and a separate setup API just to understand one test.</p> <p>Self-contained tests are easier to read because the relevant state is nearby. They are easier to review because a diff or comment contains the whole example. They are easier to change because their dependencies are explicit. They are easier to move because they carry their setup with them. They also give both humans and AI agents enough context to reason about the behaviour without reconstructing it from the rest of the file.</p> <p>A test is not better because it has fewer lines. It is better when someone can understand what it proves, why it should pass, and what would make it fail.</p> <p>Each test should own its world, and that world should be visible wherever the test appears.</p> 2026-07-22T21:27:01+03:00 2026-07-22T21:27:01+03:00 https://ryanzidago.com/posts/bylaw-a-concrete-example-of-software-engineering-guardrails/ <h1 id="introduction">Introduction</h1> <p>In the <a href="/posts/ai-coding-agents-need-software-engineering-guardrails/">previous article</a>, I argued that deterministic guardrails are a better place for repeatable engineering policies than an AI agent's context window. It avoids bloating the context window with information that could be encoded elsewhere.</p> <p>But how do you turn a policy into a guardrail?</p> <p>Start with something that can be programmatically inspected. That might be source code, an abstract syntax tree, a database schema, a query, generated HTML, or a dependency graph.</p> <p>Then:</p> <ol> <li>inspect the artifact</li> <li>recognize the pattern that matters</li> <li>express your policy as an invariant</li> <li>find a choke point through which every relevant artifact passes</li> <li>reject violations with an actionable error</li> </ol> <p>If a policy can be evaluated from data that tooling can inspect, an AI agent should not have to remember it.</p> <p>A guardrail is only reliable if it runs consistently. A single choke point lets you validate the invariant once instead of relying on every caller to remember to invoke the check.</p> <p>That is why I built <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw">Bylaw</a>. Bylaw is a collection of Elixir packages for encoding engineering policies as deterministic checks:</p> <ul> <li><a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/tree/main/packages/bylaw_credo">bylaw_credo</a> inspects source code</li> <li><a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/tree/main/packages/bylaw_postgres">bylaw_postgres</a> inspects PostgreSQL schemas</li> <li><a rel="noopener nofollow noreferrer external" target="_blank" href="https://bylaw-ecto-query.hexdocs.pm/">bylaw_ecto_query</a> inspects composed <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Query.html"><code>Ecto.Query</code></a> values</li> <li><a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/tree/main/packages/bylaw_html">bylaw_html</a> inspects HTML</li> </ul> <p>The following examples show how the same process applies to different kinds of software artifacts.</p> <h1 id="inspecting-source-code">Inspecting source code</h1> <h2 id="no-repo-transaction">No repo transaction</h2> <p><a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/blob/main/packages/bylaw_credo/lib/bylaw/credo/check/ecto/no_repo_transaction.ex"><code>Bylaw.Credo.Check.Ecto.NoRepoTransaction</code></a> prevents using the deprecated <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Repo.html#c:transaction/2"><code>Repo.transaction</code></a> and suggests using <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Repo.html#c:transact/2"><code>Repo.transact</code></a> instead.</p> <p>The artifact is Elixir source code. The pattern is a call to <code>Repo.transaction</code>. The invariant is that application code must use the current <code>Repo.transact</code> API. AI agents often use outdated APIs because older examples are well represented in their training data.</p> <p>For example, you ask an AI agent to make sure that creating an account and its audit event happen atomically. It writes:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> register_account</span><span>(</span><span>attrs</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">transaction</span><span>(</span><span class="z-keyword">fn</span><span class="z-keyword"> -&gt;</span></span> <span class="giallo-l"><span> account </span><span class="z-keyword">=</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Account</span><span>{</span><span>}</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Account</span><span>.</span><span class="z-entity z-name">changeset</span><span>(</span><span>attrs</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">insert!</span><span>(</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">insert!</span><span>(</span><span class="z-entity z-name">AuditEvent</span><span>.</span><span class="z-entity z-name">account_created</span><span>(</span><span>account</span><span>)</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> account</span></span> <span class="giallo-l"><span class="z-keyword"> end</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The implementation is otherwise correct, but the agent used <code>Repo.transaction/1</code>, an API deprecated in Ecto 3.13. The code looks plausible because years of examples use it.</p> <p>The Bylaw check rejects the call and points the agent towards the current API:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> register_account</span><span>(</span><span>attrs</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">transact</span><span>(</span><span class="z-keyword">fn</span><span class="z-keyword"> -&gt;</span></span> <span class="giallo-l"><span> account </span><span class="z-keyword">=</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Account</span><span>{</span><span>}</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Account</span><span>.</span><span class="z-entity z-name">changeset</span><span>(</span><span>attrs</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">insert!</span><span>(</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">insert!</span><span>(</span><span class="z-entity z-name">AuditEvent</span><span>.</span><span class="z-entity z-name">account_created</span><span>(</span><span>account</span><span>)</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> account</span></span> <span class="giallo-l"><span class="z-keyword"> end</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>I could add “use <code>Repo.transact</code> instead of <code>Repo.transaction</code>” to the agent's instructions, but the agent can still forget or overlook it. Once the policy is a Bylaw check, every violation produces the same result.</p> <h1 id="inspecting-a-database-schema">Inspecting a database schema</h1> <h2 id="foreign-key-actions">Foreign key actions</h2> <p>Suppose your organization's engineering standard is that foreign keys must never cascade deletes. Records should only be removed through explicit application flows, where retention requirements, audit events, and other side effects can be handled deliberately.</p> <p>The artifact is the PostgreSQL schema. The pattern is any foreign key with an <code>ON DELETE</code> action. The invariant is that every foreign key must restrict deletion while dependent records still exist.</p> <p>You encode that standard once with <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/blob/main/packages/bylaw_postgres/lib/bylaw/db/adapters/postgres/checks/foreign_key_actions.ex">Bylaw.Db.Adapters.Postgres.Checks.ForeignKeyActions</a> and apply it to the entire schema:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>{</span><span class="z-entity z-name">Bylaw</span><span>.</span><span class="z-entity z-name">Db</span><span>.</span><span class="z-entity z-name">Adapters</span><span>.</span><span class="z-entity z-name">Postgres</span><span>.</span><span class="z-entity z-name">Checks</span><span>.</span><span class="z-entity z-name">ForeignKeyActions</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> rules</span><span class="z-constant">:</span><span> [</span><span class="z-constant">on_delete</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">restrict</span><span>]</span><span>}</span></span></code></pre> <p>The schema-validation test is the choke point. It inspects the current database schema and validates every foreign key in one pass. It does not matter which migration introduced a constraint or which part of the application owns the affected tables: if the schema violates the invariant, the test fails.</p> <p>Later, an AI agent is asked to add a relationship between orders and accounts. Orders must be retained for accounting and customer support, even after an account is closed, but the agent generates this migration:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>alter </span><span class="z-entity z-name">table</span><span>(</span><span class="z-constant">:</span><span class="z-constant">orders</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> add </span><span class="z-constant">:</span><span class="z-constant">account_id</span><span>,</span></span> <span class="giallo-l"><span class="z-entity z-name"> references</span><span>(</span><span class="z-constant">:</span><span class="z-constant">accounts</span><span>,</span><span class="z-constant"> on_delete</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">delete_all</span><span>)</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> null</span><span class="z-constant">:</span><span class="z-constant"> false</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The migration looks reasonable, but deleting one account will now silently delete all of its orders. The dangerous behavior is easy to overlook because it is hidden behind the <code>on_delete</code> option. It also violates the organization's engineering standard.</p> <p>The check inspects every foreign key in the PostgreSQL schema and rejects this cascade.</p> <p>The agent then fixes the migration:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>alter </span><span class="z-entity z-name">table</span><span>(</span><span class="z-constant">:</span><span class="z-constant">orders</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> add </span><span class="z-constant">:</span><span class="z-constant">account_id</span><span>,</span></span> <span class="giallo-l"><span class="z-entity z-name"> references</span><span>(</span><span class="z-constant">:</span><span class="z-constant">accounts</span><span>,</span><span class="z-constant"> on_delete</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">restrict</span><span>)</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> null</span><span class="z-constant">:</span><span class="z-constant"> false</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Postgres will now prevent an account from being deleted while orders still reference it. The application has to handle those business records explicitly instead of losing them as a side effect.</p> <p>The same guardrail applies to every future foreign key. It protects the whole schema against drift, regardless of which tables are involved or whether the migration was written by a person or an AI agent.</p> <h1 id="inspecting-composed-queries">Inspecting composed queries</h1> <h2 id="an-ecto-query-is-a-data-structure">An Ecto query is a data structure</h2> <p>At some point, I realized that an Ecto query is a data structure.</p> <p>That changed how I thought about query validation. If a query is data, I can inspect it. If I can inspect it, I can understand what it is trying to do and define which structures are valid or invalid for my application.</p> <p>Take an ordinary query:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-entity z-name">from</span><span>(</span><span>p </span><span class="z-keyword">in</span><span class="z-entity z-name"> Post</span><span>,</span><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">post</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">|&gt;</span><span class="z-entity z-name"> join</span><span>(</span><span class="z-constant">:</span><span class="z-constant">inner</span><span>,</span><span> [</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span> a </span><span class="z-keyword">in</span><span class="z-entity z-name"> assoc</span><span>(</span><span>p</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">author</span><span>)</span><span>,</span><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">author</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">|&gt;</span><span class="z-entity z-name"> where</span><span>(</span><span>[</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span> p</span><span>.</span><span>status </span><span class="z-keyword">==</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">published</span><span class="z-punctuation z-definition z-string">&quot;</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">|&gt;</span><span class="z-entity z-name"> where</span><span>(</span><span>[</span><span class="z-constant">author</span><span class="z-constant">:</span><span> a</span><span>]</span><span>,</span><span> a</span><span>.</span><span>active </span><span class="z-keyword">==</span><span class="z-constant"> true</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">|&gt;</span><span class="z-entity z-name"> order_by</span><span>(</span><span>[</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span class="z-constant"> desc</span><span class="z-constant">:</span><span> p</span><span>.</span><span>published_at</span><span>,</span><span class="z-constant"> desc</span><span class="z-constant">:</span><span> p</span><span>.</span><span>id</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">|&gt;</span><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">all</span><span>(</span><span>)</span></span></code></pre> <p>This reads like a sequence of database operations: start from posts, join their authors, add two filters, order the results, and execute the query.</p> <p>The <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Query.html#from/2"><code>from</code></a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Query.html#join/5"><code>join</code></a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Query.html#join/5"><code>assoc</code></a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Query.html#where/3"><code>where</code></a>, and <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Query.html#order_by/3"><code>order_by</code></a> expressions build the query one step at a time.</p> <p>Before <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Repo.html#c:all/2"><code>Repo.all</code></a> executes it, those pipeline steps have built an <code>%Ecto.Query{}</code> value. Here is an abridged view of the relevant fields:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span>%</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> from</span><span class="z-constant">:</span><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">FromExpr</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> source</span><span class="z-constant">:</span><span> {</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">posts</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span><span class="z-entity z-name"> Post</span><span>}</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">post</span></span> <span class="giallo-l"><span> }</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> joins</span><span class="z-constant">:</span><span> [</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">JoinExpr</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> qual</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">inner</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> assoc</span><span class="z-constant">:</span><span> {</span><span class="z-constant">0</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">author</span><span>}</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">author</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span> ]</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> wheres</span><span class="z-constant">:</span><span> [</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">BooleanExpr</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> op</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">and</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> expr</span><span class="z-constant">:</span></span> <span class="giallo-l"><span> {</span><span class="z-constant">:</span><span class="z-constant">==</span><span>,</span><span> [</span><span>]</span><span>,</span></span> <span class="giallo-l"><span> [</span></span> <span class="giallo-l"><span> {</span><span>{</span><span class="z-constant">:</span><span class="z-constant">.</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>{</span><span class="z-constant">:</span><span class="z-constant">&amp;</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span class="z-constant">0</span><span>]</span><span>}</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">status</span><span>]</span><span>}</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">Tagged</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> value</span><span class="z-constant">:</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">published</span><span class="z-punctuation z-definition z-string">&quot;</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> type</span><span class="z-constant">:</span><span> {</span><span class="z-constant">0</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">status</span><span>}</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span> ]</span><span>}</span></span> <span class="giallo-l"><span> }</span><span>,</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">BooleanExpr</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> op</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">and</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> expr</span><span class="z-constant">:</span></span> <span class="giallo-l"><span> {</span><span class="z-constant">:</span><span class="z-constant">==</span><span>,</span><span> [</span><span>]</span><span>,</span></span> <span class="giallo-l"><span> [</span></span> <span class="giallo-l"><span> {</span><span>{</span><span class="z-constant">:</span><span class="z-constant">.</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>{</span><span class="z-constant">:</span><span class="z-constant">&amp;</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span class="z-constant">1</span><span>]</span><span>}</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">active</span><span>]</span><span>}</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">Tagged</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> value</span><span class="z-constant">:</span><span class="z-constant"> true</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> type</span><span class="z-constant">:</span><span> {</span><span class="z-constant">1</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">active</span><span>}</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span> ]</span><span>}</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span> ]</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> order_bys</span><span class="z-constant">:</span><span> [</span></span> <span class="giallo-l"><span> %</span><span class="z-entity z-name">Ecto</span><span>.</span><span class="z-entity z-name">Query</span><span>.</span><span class="z-entity z-name">ByExpr</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> expr</span><span class="z-constant">:</span><span> [</span></span> <span class="giallo-l"><span class="z-constant"> desc</span><span class="z-constant">:</span><span> {</span><span>{</span><span class="z-constant">:</span><span class="z-constant">.</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>{</span><span class="z-constant">:</span><span class="z-constant">&amp;</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span class="z-constant">0</span><span>]</span><span>}</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">published_at</span><span>]</span><span>}</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>]</span><span>}</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> desc</span><span class="z-constant">:</span><span> {</span><span>{</span><span class="z-constant">:</span><span class="z-constant">.</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>{</span><span class="z-constant">:</span><span class="z-constant">&amp;</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span class="z-constant">0</span><span>]</span><span>}</span><span>,</span><span class="z-constant"> :</span><span class="z-constant">id</span><span>]</span><span>}</span><span>,</span><span> [</span><span>]</span><span>,</span><span> [</span><span>]</span><span>}</span></span> <span class="giallo-l"><span> ]</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span> ]</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> aliases</span><span class="z-constant">:</span><span> %</span><span>{</span></span> <span class="giallo-l"><span class="z-constant"> post</span><span class="z-constant">:</span><span class="z-constant"> 0</span><span>,</span></span> <span class="giallo-l"><span class="z-constant"> author</span><span class="z-constant">:</span><span class="z-constant"> 1</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>This is Ecto's actual representation, with incidental metadata such as file names, line numbers, parameters, and subquery lists omitted. <code>&amp;0</code> refers to the root <code>Post</code> binding and <code>&amp;1</code> refers to the joined author. The important part is that the query exposes its components:</p> <ul> <li><code>from</code> identifies the base table and schema</li> <li><code>joins</code> describes the joined associations and join types</li> <li><code>wheres</code> contains the predicates and how they are combined</li> <li><code>order_bys</code> contains the fields and directions used for sorting</li> <li><code>aliases</code> maps the named bindings <code>:post</code> and <code>:author</code> to their positions in the query</li> </ul> <h2 id="the-repository-is-the-choke-point">The repository is the choke point</h2> <p>Bylaw validates the completed query through <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/ecto/Ecto.Repo.html#c:prepare_query/3"><code>Ecto.Repo.prepare_query/3</code></a>. Ecto invokes this callback for query-based repository operations, after the application has composed the query and before the repository executes it.</p> <p>That location is important. A query can be built dynamically across several functions and files. A source-code linter like <a rel="noopener nofollow noreferrer external" target="_blank" href="https://hexdocs.pm/credo/">Credo</a> can inspect each expression independently, but it may never see the final query shape. <code>prepare_query/3</code> receives the composed <code>%Ecto.Query{}</code> at the shared repository boundary, giving Bylaw one place to validate the relevant queries before they reach the database.</p> <p>This means a program can ask questions about the query before it reaches the database:</p> <ul> <li>Does it contain contradictory predicates?</li> <li>Can its ordering produce non-deterministic results?</li> <li>Does it join a table that this operation should not access?</li> <li>Is a required tenant or authorization filter present?</li> <li>Does an update or delete have an appropriately narrow filter?</li> </ul> <p>Once I saw the query as data, guardrails around Ecto queries felt like a natural consequence. Bylaw can inspect the same structure that Ecto will execute, recognize unsafe or invalid patterns, and reject the query before it reaches the database.</p> <h2 id="conflicting-where-predicates">Conflicting where predicates</h2> <p>Suppose your application builds a query incrementally across multiple files.</p> <p>The artifact is the composed Ecto query. The pattern is multiple equality predicates on the same field. The invariant is that those predicates cannot require mutually exclusive values.</p> <p>The posts context starts from a scope that only returns published posts:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"> lib/my_app/blog.ex</span></span> <span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> list_public_posts</span><span>(</span><span>params</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-entity z-name"> from</span><span>(</span><span>p </span><span class="z-keyword">in</span><span class="z-entity z-name"> Post</span><span>,</span><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">post</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> where</span><span>(</span><span>[</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span> p</span><span>.</span><span>status </span><span class="z-keyword">==</span><span class="z-constant"> :</span><span class="z-constant">published</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> PostFilters</span><span>.</span><span class="z-entity z-name">apply</span><span>(</span><span>params</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">all</span><span>(</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Later, an AI agent is asked to add a status filter to the API. It works in a separate filter module and writes:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#</span><span class="z-comment"> lib/my_app/blog/post_filters.ex</span></span> <span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> apply</span><span>(</span><span>query</span><span>,</span><span> %</span><span>{</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">status</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> =&gt;</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">draft</span><span class="z-punctuation z-definition z-string">&quot;</span><span>}</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-entity z-name"> where</span><span>(</span><span>query</span><span>,</span><span> [</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span> p</span><span>.</span><span>status </span><span class="z-keyword">==</span><span class="z-constant"> :</span><span class="z-constant">draft</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> apply</span><span>(</span><span>query</span><span>,</span><span class="z-comment"> _params</span><span>)</span><span>,</span><span class="z-constant"> do:</span><span> query</span></span></code></pre> <p>Each function looks reasonable when read on its own. But a request with <code>?status=draft</code> produces this query:</p> <pre class="giallo z-code"><code data-lang="sql"><span class="giallo-l"><span class="z-keyword">WHERE</span><span class="z-keyword"> status</span><span class="z-keyword"> =</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">published</span><span class="z-punctuation z-definition z-string">&#39;</span><span class="z-keyword"> AND</span><span class="z-keyword"> status</span><span class="z-keyword"> =</span><span class="z-punctuation z-definition z-string"> &#39;</span><span class="z-string">draft</span><span class="z-punctuation z-definition z-string">&#39;</span></span></code></pre> <p>The query does not crash. It silently returns no records. Because the predicates were added in different files, inspecting either function by itself will not reveal the bug.</p> <p><a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/blob/main/packages/bylaw_ecto_query/lib/bylaw/ecto/query/checks/conflicting_where_predicates.ex">Bylaw.Ecto.Query.Checks.ConflictingWherePredicates</a> inspects the composed query and reports the conflicting predicates.</p> <h2 id="deterministic-order">Deterministic order</h2> <p>Another query invariant is deterministic ordering.</p> <p>The artifact is again the composed Ecto query. The pattern is a query that returns records in a non-deterministic order. The invariant is that an ordered result set must have a stable order across executions.</p> <p>An AI agent writes an API endpoint:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> index</span><span>(</span><span>conn</span><span>,</span><span class="z-comment"> _params</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> posts </span><span class="z-keyword">=</span></span> <span class="giallo-l"><span class="z-entity z-name"> from</span><span>(</span><span>p </span><span class="z-keyword">in</span><span class="z-entity z-name"> Post</span><span>,</span><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">post</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> order_by</span><span>(</span><span>[</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span class="z-constant"> desc</span><span class="z-constant">:</span><span> p</span><span>.</span><span>inserted_at</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">all</span><span>(</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name"> json</span><span>(</span><span>conn</span><span>,</span><span> %</span><span>{</span><span class="z-constant">data</span><span class="z-constant">:</span><span> posts</span><span>}</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Two posts can have the same <code>inserted_at</code> value. When that happens, Postgres is free to return those posts in a different order between requests.</p> <p>This is based on a real issue I encountered at work. The bug itself was small: an API endpoint returned a list without deterministic ordering. It passed through tests and code review, was deployed, and eventually affected a customer consuming the API.</p> <p>Once a bug like this reaches a customer, fixing one missing tie-breaker can consume time across an entire organization:</p> <ol> <li>The customer notices inconsistent results and investigates whether the problem is in their own integration.</li> <li>The problem has to bother them enough that they spend more time writing a support ticket.</li> <li>Support investigates whether the issue has already been reported, fixed, or planned.</li> <li>If it has not, support creates an actionable item for product.</li> <li>Product evaluates the issue and prioritizes it against other work.</li> <li>The issue is assigned to a developer, who reproduces it, identifies the missing deterministic order, writes a regression test, and implements the fix.</li> <li>The change goes through code review and deployment.</li> <li>Engineering informs support that the fix has been released.</li> <li>Support informs the customer, who then verifies the behavior in their integration.</li> </ol> <p>Every step is reasonable, but together they consume significant time from the customer, support, product, and engineering. The organization pays that cost because a small invariant was not enforced before release.</p> <p>Fixing this endpoint does not prevent the same mistake from happening again. A regression test can protect this particular endpoint, but another developer or AI agent can add a different query without deterministic ordering. That query can also pass through tests and code review, reach production, and start the same lifecycle again.</p> <p><a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ryanzidago/bylaw/blob/main/packages/bylaw_ecto_query/lib/bylaw/ecto/query/checks/deterministic_order.ex">Bylaw.Ecto.Query.Checks.DeterministicOrder</a> requires the root schema's primary key:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> index</span><span>(</span><span>conn</span><span>,</span><span class="z-comment"> _params</span><span>)</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> posts </span><span class="z-keyword">=</span></span> <span class="giallo-l"><span class="z-entity z-name"> from</span><span>(</span><span>p </span><span class="z-keyword">in</span><span class="z-entity z-name"> Post</span><span>,</span><span class="z-constant"> as</span><span class="z-constant">:</span><span class="z-constant"> :</span><span class="z-constant">post</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> order_by</span><span>(</span><span>[</span><span class="z-constant">post</span><span class="z-constant">:</span><span> p</span><span>]</span><span>,</span><span class="z-constant"> desc</span><span class="z-constant">:</span><span> p</span><span>.</span><span>inserted_at</span><span>,</span><span class="z-constant"> desc</span><span class="z-constant">:</span><span> p</span><span>.</span><span>id</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword"> |&gt;</span><span class="z-entity z-name"> Repo</span><span>.</span><span class="z-entity z-name">all</span><span>(</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name"> json</span><span>(</span><span>conn</span><span>,</span><span> %</span><span>{</span><span class="z-constant">data</span><span class="z-constant">:</span><span> posts</span><span>}</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>The primary key acts as a tie-breaker. Even when two posts have the same <code>inserted_at</code> value, their order is now stable between API requests.</p> <p>Encoding this rule as a guardrail catches the missing primary key while the query is being developed. The agent or developer fixes it immediately, before the bug can begin that lifecycle. Because the check runs at the repository choke point, it protects future queries as well as the endpoint that originally exposed the problem. This is the practical value of guardrails: they preserve attention across the organization by preventing known classes of mistakes at the cheapest point to correct them.</p> <p>Another field with a unique database constraint could also make the ordering deterministic. However, an Ecto schema does not expose arbitrary unique indexes, so this query check cannot prove that such a field is unique. It only accepts the root schema's primary key, which it can verify through Ecto schema reflection. A database-aware check could inspect unique constraints and safely recognize additional tie-breakers.</p> <h2 id="a-deliberate-trade-off">A deliberate trade-off</h2> <p>Ecto exposes the <code>%Ecto.Query{}</code> struct, but the internal representation of its expressions is not a stable extension API. A future Ecto release could change that representation and require Bylaw's query checks to be updated.</p> <p>I decided that trade-off was acceptable for three reasons:</p> <ul> <li>Ecto's query API has been fairly stable across releases</li> <li>I run these checks in development and test, not in production</li> <li>if an Ecto upgrade breaks a check, it fails in the development lifecycle instead of changing behavior for customers</li> </ul> <p>The checks may need maintenance after an Ecto upgrade, but that failure should be visible and relatively easy to fix before deployment.</p> <p>The checks are also intentionally narrow. They do not attempt to prove every semantic property of a query. Each check recognizes a specific set of query shapes and may need to balance false positives against false negatives. Their purpose is to steer the application toward explicit, approved patterns and catch high-value mistakes consistently.</p> <h1 id="what-makes-a-good-guardrail">What makes a good guardrail?</h1> <p>The examples above inspect different artifacts, but the guardrails share the same properties:</p> <ul> <li><strong>Deterministic:</strong> the same artifact always produces the same result</li> <li><strong>Focused:</strong> each check enforces one policy</li> <li><strong>Fast:</strong> developers and agents receive feedback while the relevant change is still fresh</li> <li><strong>Actionable:</strong> the error explains what failed and how to correct it</li> <li><strong>Hard to bypass accidentally:</strong> the checks run before a change is merged</li> </ul> <p>Guardrails are also educational tools for humans and AI agents. A developer may not know that a database does not guarantee row order without an <code>ORDER BY</code>, or that ordering by a non-unique field can still produce unstable results. When a guardrail rejects the query, it can identify the violated invariant, explain why the query is unsafe, and show what needs to change.</p> <p>The timing makes that feedback especially effective. The human or agent receives the explanation while writing the relevant code, with the failing query directly in front of them. The lesson is attached to a concrete decision instead of presented as an abstract rule that they may need someday.</p> <p>A company-wide <code>CLAUDE.md</code>, style guide, or prompt can document the same policy, but every contributor and agent still has to find, interpret, and remember it. An actionable guardrail delivers the relevant policy at the point of violation and prevents the code from moving forward until the issue is addressed. It enforces the invariant and teaches the reasoning behind it in the same feedback loop.</p> <p>A noisy or ambiguous check will eventually be ignored. A useful guardrail identifies a real violation, explains why it matters, and gives the developer or agent a clear path forward.</p> <h1 id="where-to-enforce-guardrails">Where to enforce guardrails</h1> <p>A guardrail is most effective when it runs throughout the development lifecycle:</p> <ul> <li>in the agent's feedback loop after it changes code</li> <li>before commits or pushes</li> <li>in CI before merge</li> </ul> <p>Fast local feedback helps the agent correct its own output. CI provides the final guarantee that the policy applies to every contribution.</p> <h1 id="conclusion">Conclusion</h1> <p>Building a software engineering guardrail starts with an inspectable artifact. Recognize the pattern that carries risk, express the desired policy as an invariant, and enforce it at a choke point through which every relevant artifact passes.</p> <p>Bylaw applies that process to Elixir source code, PostgreSQL schemas, Ecto queries, and HTML. The result is a set of policies that AI agents and developers do not need to remember because the tooling enforces them every time.</p> 2026-07-22T21:26:59+03:00 2026-07-22T21:26:59+03:00 https://ryanzidago.com/posts/ai-coding-agents-need-software-engineering-guardrails/ <h1 id="problem">Problem</h1> <p>LLMs cannot reliably remember every rule in your codebase:</p> <ul> <li>your coding style guide</li> <li>how to write a database query returning multiple records in a deterministic order</li> <li>add type specs to functions</li> <li>soft deleting a record instead of hard deleting it</li> <li>etc ...</li> </ul> <p>Every time you tell an LLM to remember a rule, you're paying for that rule on every generation. The model may forget it, misunderstand it, or decide another instruction is more important. It is very tempting to pack a huge list of instructions into the LLM's prompt or even a review agent but:</p> <ul> <li>it is slow</li> <li>it is expensive</li> <li>and above all, it is non-deterministic.</li> </ul> <p>Engineering rules are different. Once you've decided that all public functions need typespecs, or that queries should always have deterministic ordering, there's little value in asking an AI agent to rediscover those rules every time it writes code.</p> <p>Which is why, with LLMs, I always ask myself:</p> <blockquote> <p>Can this probabilistic output be made deterministic so producing the output is faster, cheaper, and more reliable?</p> </blockquote> <h1 id="solution">Solution</h1> <p>Encode engineering policies as deterministic guardrails enforced by:</p> <ul> <li>static analysis and type checkers (<a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/erlang/otp/tree/master/lib/dialyzer">Dialyzer</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/microsoft/pyright">Pyright</a>)</li> <li>linters and structural code checks (<a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/rrrene/credo">Credo</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/eslint/eslint">ESLint</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/ast-grep/ast-grep">ast-grep</a>)</li> <li>formatters (<a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/elixir-lang/elixir/blob/main/lib/mix/lib/mix/tasks/format.ex">mix format</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/prettier/prettier">Prettier</a>)</li> <li>test frameworks (<a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/elixir-lang/elixir/tree/main/lib/ex_unit">ExUnit</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/vitest-dev/vitest">Vitest</a>)</li> <li>security checks (<a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/nccgroup/sobelow">Sobelow</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/semgrep/semgrep">Semgrep</a>)</li> <li>architectural checks (<a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/sasa1977/boundary">Boundary</a>)</li> </ul> <p>Each tool encodes a different kind of rule. Formatters eliminate debates about syntax. Type checkers and static analyzers catch invalid assumptions. Linters and structural code checks can enforce project-specific conventions and code patterns. Tests protect behavioral invariants. Security checks detect known vulnerabilities and unsafe patterns. Architectural checks enforce dependencies and boundaries between components.</p> <p>Instead of asking AI to remember your policies, configure tools that enforce them consistently.</p> <p>Ensure those guardrails run throughout the software development lifecycle:</p> <ul> <li>prevent commits and pushes when a guardrail fails</li> <li>fail CI when a guardrail fails</li> </ul> <p>When a policy is violated, its error message should clearly explain what policy was violated, why it exists, and whether bypassing it is ever appropriate.</p> <h1 id="conclusion">Conclusion</h1> <p>If you find yourself repeatedly reminding an AI agent about the same engineering rule over again and again, stop and ask yourself if you could encode that rule as a deterministic guardrail. In the next article, we will look at how to build them.</p> 2026-07-17T19:30:00+03:00 2026-07-17T19:30:00+03:00 https://ryanzidago.com/posts/reducing-elixir-worktree-setup-time-by-83-percent/ <h2 id="tl-dr">TL;DR</h2> <ul> <li>Copying <code>deps</code> and <code>_build</code> is the obvious first optimization. How much it helps depends on the project's build profile: dependency-heavy projects benefit most, while projects dominated by application compilation may still have substantial work remaining.</li> <li>Mix may still recompile the application's own modules. In Phoenix, that happened because the compiler manifest recorded the original checkout's current working directory.</li> <li>On Elixir 1.19.2 or newer, <code>elixirc_options: [check_cwd: false]</code> can remove this blanket invalidation, but only when the project build is otherwise relocatable.</li> <li>Copy-on-write cloning keeps each worktree independently writable without immediately making a full second copy of the cached files.</li> <li>Always run <code>mix deps.get</code> and <code>mix compile</code> after cloning to validate your changes.</li> <li>In my Phoenix benchmark, copying the caches reduced setup from 12.01 seconds to 2.47 seconds. Making the project build relocatable and cloning copy-on-write reduced it further to 2.06 seconds.</li> </ul> <h2 id="the-problem">The problem</h2> <p>Creating a <a rel="noopener nofollow noreferrer external" target="_blank" href="https://git-scm.com/docs/git-worktree">Git worktree</a> took milliseconds. Preparing it for Elixir development took much longer because the new checkout had no <code>deps</code> or <code>_build</code>.</p> <p>The obvious solution was to copy both directories from an existing checkout. It worked: Mix reused the compiled dependencies and setup became much faster. But <strong>Phoenix still recompiled all 74 of its own project files, even when the worktree started at the same commit</strong> and the copy preserved timestamps. Why is that?</p> <h2 id="the-solution">The solution</h2> <p>I benchmarked <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/phoenixframework/phoenix">Phoenix</a> at commit <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/phoenixframework/phoenix/commit/0d9c79ebfc0f2065b41e268551638bd8b767eda9"><code>0d9c79e</code></a> while answering that question.</p> <p>Assume my Mix project is already compiled on <code>main</code>:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>/Users/ryanzidago/Projects/my_mix_project</span></span></code></pre> <p>I create a worktree from the same commit:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-support">cd</span><span class="z-string"> /Users/ryanzidago/Projects/my_mix_project</span></span> <span class="giallo-l"><span class="z-entity z-name">git</span><span class="z-string"> worktree</span><span class="z-string"> add</span><span class="z-string"> .worktrees/add-export</span><span class="z-constant"> -</span><span class="z-constant">b</span><span class="z-string"> add-export</span><span class="z-string"> main</span></span></code></pre> <p>The worktree lives at a different absolute path:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>/Users/ryanzidago/Projects/my_mix_project/.worktrees/add-export</span></span></code></pre> <p>Git does not put ignored build artifacts in the new checkout. Instead of rebuilding them from scratch, I seed the worktree from <code>main</code> and let Mix validate them:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">cp</span><span class="z-constant"> -</span><span class="z-constant">a</span><span class="z-string"> /Users/ryanzidago/Projects/my_mix_project/deps/.</span><span class="z-constant"> \</span></span> <span class="giallo-l"><span class="z-string"> /Users/ryanzidago/Projects/my_mix_project/.worktrees/add-export/deps/</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">cp</span><span class="z-constant"> -</span><span class="z-constant">a</span><span class="z-string"> /Users/ryanzidago/Projects/my_mix_project/_build/.</span><span class="z-constant"> \</span></span> <span class="giallo-l"><span class="z-string"> /Users/ryanzidago/Projects/my_mix_project/.worktrees/add-export/_build/</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-support">cd</span><span class="z-string"> /Users/ryanzidago/Projects/my_mix_project/.worktrees/add-export</span></span> <span class="giallo-l"><span class="z-entity z-name">mix</span><span class="z-string"> deps.get</span></span> <span class="giallo-l"><span class="z-entity z-name">mix</span><span class="z-string"> compile</span></span></code></pre> <p><code>deps</code> contains dependency source. <code>_build</code> contains compiled dependencies, project modules, and compiler manifests. Copying both provided the largest improvement because Mix could reuse the compiled dependency tree. However, Phoenix's own modules still recompiled:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>Compiling 74 files (.ex)</span></span> <span class="giallo-l"><span>Generated phoenix app</span></span></code></pre> <p>Even with that recompilation, setup fell from 12.01 seconds to 2.47 seconds. Copying the caches saved 9.54 seconds by preserving the compiled dependencies, but it did not yet preserve Phoenix's own compiled modules.</p> <p>Why does the project still recompile?</p> <p>The explanation was inside <code>_build</code>. Mix writes compiler manifests describing the inputs and assumptions behind compiled modules, and the Elixir manifest stores the current working directory. Copying <code>_build</code> preserved the path from <code>main</code>:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>main</span></span> <span class="giallo-l"><span>cwd = /Users/ryanzidago/Projects/my_mix_project</span></span> <span class="giallo-l"><span>manifest stores this cwd</span></span> <span class="giallo-l"><span> │</span></span> <span class="giallo-l"><span> │ copy deps + _build</span></span> <span class="giallo-l"><span> ▼</span></span> <span class="giallo-l"><span>worktree</span></span> <span class="giallo-l"><span>cwd = /Users/ryanzidago/Projects/my_mix_project/.worktrees/add-export</span></span> <span class="giallo-l"><span>manifest still stores the main cwd</span></span> <span class="giallo-l"><span> │</span></span> <span class="giallo-l"><span> ▼</span></span> <span class="giallo-l"><span>stored cwd != current cwd → manifest stale → recompile project</span></span></code></pre> <p>With the default <code>check_cwd: true</code>, Mix deliberately treats that mismatch as stale. The behavior is visible in <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/elixir-lang/elixir/blob/v1.20.2/lib/mix/lib/mix/compilers/elixir.ex"><code>Mix.Compilers.Elixir</code></a>: it writes <code>File.cwd!()</code> into the manifest and compares it on the next compile.</p> <p>That check is a correctness feature. Compiled modules retain absolute source-path metadata from the original checkout. Relocating them does not normally change runtime behavior, but developer tools that use this metadata may still point to the original source tree. Recompiling updates the metadata to the new checkout.</p> <p>More importantly, application code can use the cwd during compilation:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">defmodule</span><span class="z-entity"> MyMixProject</span><span>.</span><span class="z-entity z-name">BuildInfo</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span class="z-variable z-other"> @</span><span class="z-variable z-other">project_root</span><span class="z-entity z-name"> File</span><span>.</span><span class="z-entity z-name">cwd!</span><span>(</span><span>)</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword"> def</span><span class="z-entity z-name"> project_root</span><span>,</span><span class="z-constant"> do:</span><span class="z-variable z-other"> @</span><span class="z-variable z-other">project_root</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p><code>@project_root</code> becomes part of the compiled module. Copying that BEAM into a worktree without recompiling it leaves runtime code pointing to <code>main</code>.</p> <p>There are therefore two kinds of path dependence:</p> <ol> <li><strong>Metadata-only:</strong> the BEAM points to source on <code>main</code>. Runtime behavior is unchanged, but tooling may open the wrong checkout.</li> <li><strong>Behavioral:</strong> compile-time code embeds a path or data read through it. Reusing the BEAM can be incorrect.</li> </ol> <p>Most application modules do not behave differently based on their checkout path. The risky cases are compile-time macros and module attributes that read files relative to the project root, custom Mix compilers, code reloaders, absolute paths in compile-time configuration, and native builds with absolute include or runtime paths. <code>check_cwd: false</code> accepts the metadata tradeoff, so do not use it while behavioral path dependence remains implicit.</p> <p>Elixir 1.19.2 introduced a configurable cwd check. If the remaining project recompilation matters, audit that path usage before adding this to <code>mix.exs</code>:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> project</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> [</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> #</span><span class="z-comment"> ...</span></span> <span class="giallo-l"><span class="z-constant"> elixirc_options</span><span class="z-constant">:</span><span> [</span><span class="z-constant">check_cwd</span><span class="z-constant">:</span><span class="z-constant"> false</span><span>]</span></span> <span class="giallo-l"><span> ]</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Changing the option changes the compiler cache key, so build <code>main</code> once after adding it. New worktrees can then reuse manifests created with the same compiler options.</p> <p>Setting a literal <code>false</code> makes relocation a project-wide property. To opt in only for the worktree workflow, map the option to <code>MIX_CHECK_CWD</code>:</p> <pre class="giallo z-code"><code data-lang="elixir"><span class="giallo-l"><span class="z-keyword">def</span><span class="z-entity z-name"> project</span><span class="z-keyword"> do</span></span> <span class="giallo-l"><span> check_cwd? </span><span class="z-keyword">=</span><span class="z-entity z-name"> System</span><span>.</span><span class="z-entity z-name">get_env</span><span>(</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string">MIX_CHECK_CWD</span><span class="z-punctuation z-definition z-string">&quot;</span><span>)</span><span class="z-keyword"> !=</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">false</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> [</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment"> #</span><span class="z-comment"> ...</span></span> <span class="giallo-l"><span class="z-constant"> elixirc_options</span><span class="z-constant">:</span><span> [</span><span class="z-constant">check_cwd</span><span class="z-constant">:</span><span> check_cwd?</span><span>]</span></span> <span class="giallo-l"><span> ]</span></span> <span class="giallo-l"><span class="z-keyword">end</span></span></code></pre> <p>Worktree commands can then run with <code>MIX_CHECK_CWD=false</code>, while teammates and CI retain the default. Only the literal value <code>false</code> disables the check. The base cache and worktree must use the same value because <code>elixirc_options</code> participates in the compiler cache key. Warm the base cache once with <code>MIX_CHECK_CWD=false mix compile</code>, then export the same value in the setup hook.</p> <p>Phoenix still compiled four files because they consult <code>Mix.Project</code> at compile time:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>recompiled 4 of 74 project files:</span></span> <span class="giallo-l"><span> lib/mix/tasks/phx.ex</span></span> <span class="giallo-l"><span> lib/mix/tasks/phx.gen.embedded.ex</span></span> <span class="giallo-l"><span> lib/mix/tasks/phx.gen.schema.ex</span></span> <span class="giallo-l"><span> lib/phoenix/code_reloader/server.ex</span></span></code></pre> <p>Setup fell again, from 2.47 seconds to 2.17 seconds, while project recompilation dropped from 74 files to four. The option removes blanket cwd invalidation; Mix continues applying its other stale checks.</p> <p>The last optimization is to clone the caches copy-on-write. On <a rel="noopener nofollow noreferrer external" target="_blank" href="https://developer.apple.com/documentation/foundation/about-apple-file-system">APFS</a>, <code>cp -a -c</code> creates file clones. On a reflink-capable Linux filesystem such as <a rel="noopener nofollow noreferrer external" target="_blank" href="https://btrfs.readthedocs.io/en/latest/Reflink.html">btrfs</a> or <a rel="noopener nofollow noreferrer external" target="_blank" href="https://docs.kernel.org/filesystems/xfs/index.html">XFS</a>, GNU <a rel="noopener nofollow noreferrer external" target="_blank" href="https://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html"><code>cp --reflink</code></a> provides the equivalent behavior. Each worktree gets an independently writable directory tree, while unchanged data initially shares physical storage. Regular <code>cp -a</code> still provides the main compilation benefit; copy-on-write only reduces the time and storage needed to seed the cache. In the benchmark, it reduced setup from 2.17 seconds to 2.06 seconds.</p> <p>I use <a rel="noopener nofollow noreferrer external" target="_blank" href="https://worktrunk.dev/">Worktrunk</a> to run the complete hook when it creates a worktree, but the script works with any Git worktree workflow. It includes the optional cwd optimization; remove the <code>MIX_CHECK_CWD</code> export if you only want the broadly applicable cache-copying benefit:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-support">set</span><span class="z-constant"> -</span><span class="z-constant">eu</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-variable z-other">base_worktree</span><span class="z-keyword">=</span><span class="z-variable">$</span><span class="z-variable">1</span></span> <span class="giallo-l"><span class="z-storage">export</span><span class="z-variable z-other"> MIX_CHECK_CWD</span><span class="z-keyword">=</span><span class="z-variable z-other">false</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">clone_directory</span><span>(</span><span>)</span><span> {</span></span> <span class="giallo-l"><span class="z-variable z-other"> source_path</span><span class="z-keyword">=</span><span class="z-variable">$</span><span class="z-variable">1</span></span> <span class="giallo-l"><span class="z-variable z-other"> destination_path</span><span class="z-keyword">=</span><span class="z-variable">$</span><span class="z-variable">2</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> [</span><span class="z-keyword"> -d</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">source_path</span><span class="z-punctuation z-definition z-string">&quot;</span><span> ]</span><span class="z-keyword"> |</span><span class="z-keyword">|</span><span class="z-keyword"> return</span><span class="z-constant"> 0</span></span> <span class="giallo-l"><span class="z-entity z-name"> mkdir</span><span class="z-constant"> -</span><span class="z-constant">p</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">destination_path</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword"> if</span><span> [</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-string">$(</span><span class="z-entity z-name">uname</span><span class="z-constant"> -</span><span class="z-constant">s</span><span class="z-string">)</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-keyword"> =</span><span> Darwin </span><span>]</span><span>;</span><span class="z-keyword"> then</span></span> <span class="giallo-l"><span class="z-entity z-name"> cp</span><span class="z-constant"> -</span><span class="z-constant">a</span><span class="z-constant"> -</span><span class="z-constant">c</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">source_path</span><span class="z-string">/.</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">destination_path</span><span class="z-string">/</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"><span class="z-keyword"> elif</span><span class="z-keyword"> !</span><span class="z-entity z-name"> cp</span><span class="z-constant"> -</span><span class="z-constant">a</span><span class="z-constant"> -</span><span class="z-constant">-reflink=always</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">source_path</span><span class="z-string">/.</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">destination_path</span><span class="z-string">/</span><span class="z-punctuation z-definition z-string">&quot;</span><span>;</span><span class="z-keyword"> then</span></span> <span class="giallo-l"><span class="z-entity z-name"> cp</span><span class="z-constant"> -</span><span class="z-constant">a</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">source_path</span><span class="z-string">/.</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">destination_path</span><span class="z-string">/</span><span class="z-punctuation z-definition z-string">&quot;</span></span> <span class="giallo-l"><span class="z-keyword"> fi</span></span> <span class="giallo-l"><span>}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">clone_directory</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">base_worktree</span><span class="z-string">/deps</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string"> deps</span></span> <span class="giallo-l"><span class="z-entity z-name">clone_directory</span><span class="z-punctuation z-definition z-string"> &quot;</span><span class="z-variable z-other">$</span><span class="z-variable z-other">base_worktree</span><span class="z-string">/_build</span><span class="z-punctuation z-definition z-string">&quot;</span><span class="z-string"> _build</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">mix</span><span class="z-string"> deps.get</span></span> <span class="giallo-l"><span class="z-entity z-name">mix</span><span class="z-string"> compile</span></span></code></pre> <p>Always run <code>mix deps.get</code> and <code>mix compile</code> after cloning. The first command reconciles the copied dependency source with the worktree's <code>mix.lock</code>; the second recompiles anything genuinely stale. If compilation fails, report the error instead of automatically deleting <code>_build</code>: the branch may contain a real source error, and a full rebuild would only repeat it more slowly.</p> <p>On my machine, the final median consisted of 53ms to create the worktree, 215ms to clone the caches, 1.33s to reconcile dependencies, and 441ms to compile.</p> <p>Here is the complete progression:</p> <table><thead><tr><th>Setup</th><th style="text-align: right">Median</th><th style="text-align: right">Range</th><th style="text-align: right">Phoenix files compiled</th><th style="text-align: right">Saved from previous step</th></tr></thead><tbody> <tr><td>Fresh worktree</td><td style="text-align: right">12.01s</td><td style="text-align: right">10.53–13.98s</td><td style="text-align: right">74</td><td style="text-align: right">N/A</td></tr> <tr><td>Copy <code>deps</code> and <code>_build</code></td><td style="text-align: right">2.47s</td><td style="text-align: right">2.29–2.73s</td><td style="text-align: right">74</td><td style="text-align: right">9.54s</td></tr> <tr><td>Add <code>check_cwd: false</code></td><td style="text-align: right">2.17s</td><td style="text-align: right">2.13–2.27s</td><td style="text-align: right">4</td><td style="text-align: right">0.30s</td></tr> <tr><td>Copy-on-write clone</td><td style="text-align: right">2.06s</td><td style="text-align: right">1.92–2.18s</td><td style="text-align: right">4</td><td style="text-align: right">0.11s</td></tr> </tbody></table> <p>Copying the caches delivered 96% of the final 9.95-second saving. Making the project build relocatable and cloning copy-on-write recovered the smaller remaining opportunity.</p> <p>Every scenario includes <code>git worktree add</code>, <code>mix deps.get</code>, and <code>mix compile</code>. The benchmark used one discarded warmup and five measured trials on an Apple M4 Max, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://developer.apple.com/documentation/foundation/about-apple-file-system">APFS</a>, Erlang/OTP 29, and Elixir 1.20.2.</p> <h2 id="the-learning">The learning</h2> <p>Build caches are not just collections of files. They encode assumptions about the environment in which they were created: paths, toolchain versions, operating systems, architectures, configuration, and dependency state.</p> <p>Whenever you relocate a cache, the useful question is not only “Can I copy these files?” It is “Which assumptions were encoded when they were built, and are those assumptions still true?”</p> <p>I repeated the experiment against <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/livebook-dev/livebook">Livebook</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/plausible/analytics">Plausible Analytics</a>, <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/hexpm/hexpm">Hex.pm</a>, and <a rel="noopener nofollow noreferrer external" target="_blank" href="https://github.com/supabase/realtime">Supabase Realtime</a>. Copying the caches reduced median setup time by 79% to 91% even though every application file still recompiled. Those projects embed checkout-specific absolute paths in compile-time configuration, so disabling the manifest cwd check was not enough to make their builds relocatable. Phoenix was the controlled example because it isolated the cwd check cleanly.</p> <p>This also means the benefit depends on the project. A dependency-heavy project with little application code can gain most of its speedup just by copying the caches. A project dominated by expensive application compilation may have much more work remaining.</p> <p>At minimum, reuse caches only across compatible:</p> <ul> <li>commits and source trees</li> <li><code>MIX_ENV</code> values and <code>mix.lock</code> files</li> <li>compiler options</li> <li>Elixir and Erlang/OTP versions</li> <li>operating systems and CPU architectures, especially with native dependencies</li> </ul> <p>Clone into empty cache directories, preserve timestamps, and keep every worktree independently writable. Do not symlink every worktree to one shared build directory: branches would become competing owners of the same modules and manifests.</p> <p>The goal is not to bypass Mix. It is to give Mix a cheap, independent cache that it can validate. Once the cache's assumptions become explicit, reuse stops being only a file-copying question and becomes a relocatability question, with Mix remaining the correctness gate.</p> <p>The <a href="https://gist.github.com/ryanzidago/d943ca563a00a528874338337ceb91db" target="_blank" rel="noopener noreferrer">reproduction script</a> pins the Phoenix commit, prints every phase, reports its environment, and verifies the result with module loading and Phoenix router tests. The benchmark used a warm Hex cache and excluded network download time, so treat it as one-machine evidence rather than a promise of equivalent timings elsewhere.</p> 2026-03-28T17:20:22+02:00 2026-03-28T17:20:22+02:00 https://ryanzidago.com/posts/what-i-learned-shipping-a-visual-dashboard-builder-in-3-days/ <p>A customer wanted a visual dashboard builder (where you click buttons and create your own dashboard). They needed this since yesterday™, but I absolutely didn't want to compromise on quality as I knew having this poorly implemented would bite us a lot in the future.</p> <p>In 3 days we shipped a usable first version of a visual dashboard builder where users could:</p> <ul> <li>compose a query by selecting a source, dimensions, measures, filters, sorts, limit, and timezone</li> <li>use the result of that query to configure a widget, including chart settings and previewing the output</li> <li>place one or more widgets into a dashboard grid, then drag, drop, and resize them until the layout felt right</li> <li>save, edit, and delete dashboards once they were happy with the result</li> </ul> <p>Under the hood, each widget stored structured query intent rather than relying on SQL as the editing format. That made the builder easier to reopen, validate, and evolve over time. The system could still compile that intent into executable ClickHouse SQL for preview and persistence, but the source of truth for editing stayed in a format the UI could understand directly.</p> <p>I decided to organise the work in 4 phases:</p> <ul> <li>research planning</li> <li>architecture planning</li> <li>implementation planning</li> <li>implementation</li> </ul> <p>During this work I learned that:</p> <ul> <li>In my experience on this project, Codex was amazing at research and following a plan but weak at UI work</li> <li>In my experience on this project, Claude Code did decent research, could follow a plan, and was much stronger at UI work</li> <li>As such, from now on I shall remember:</li> </ul> <blockquote> <p>Switching large language model is faster than forcing it to do something it isn't good at</p> </blockquote> <ul> <li>It is hard to know in advance which model will be best at a given task, so I use several and keep the strongest output</li> <li>User interfaces deserve their own spec (which can be a .html file)</li> </ul> <p>Here's how I proceeded in more detail:</p> <h1 id="research-plan">Research plan</h1> <p>First, I asked an LLM to produce a research prompt:</p> <pre class="giallo z-code"><code data-lang="markdown"><span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Context</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Our tech stack</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> What the dashboard builder needs to do</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> What I need from this research</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 1. Architecture </span><span class="z-entity z-name">&amp;</span><span class="z-entity z-name"> design patterns</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 2. ClickHouse-specific challenges</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 3. Phoenix LiveView-specific considerations</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 4. Common bugs </span><span class="z-entity z-name">&amp;</span><span class="z-entity z-name"> failure modes</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 5. Query safety </span><span class="z-entity z-name">&amp;</span><span class="z-entity z-name"> guardrails</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 6. Dashboard layout </span><span class="z-entity z-name">&amp;</span><span class="z-entity z-name"> composition</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 7. Lessons from existing tools</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> 8. Build vs. integrate</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Output format</span></span> <span class="giallo-l"><span>Please structure your response with clear sections matching the numbered topics above. For each section:</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Lead with the </span><span class="z-markup z-bold">**</span><span class="z-markup z-bold">key insight or recommendation</span><span class="z-markup z-bold">**</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Follow with </span><span class="z-markup z-bold">**</span><span class="z-markup z-bold">supporting evidence</span><span class="z-markup z-bold">**</span><span> (how other tools do it, common patterns)</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> End with </span><span class="z-markup z-bold">**</span><span class="z-markup z-bold">specific pitfalls to avoid</span><span class="z-markup z-bold">**</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Where relevant, suggest </span><span class="z-markup z-bold">**</span><span class="z-markup z-bold">concrete Elixir/LiveView patterns</span><span class="z-markup z-bold">**</span><span> or pseudocode</span></span></code></pre> <p>Then I asked the following LLM platforms to produce their own research outputs:</p> <ul> <li>claude code (cli)</li> <li>codex (cli)</li> <li>claude (web)</li> <li>gemini (web)</li> <li>perplexity (web)</li> </ul> <p>After that, I asked Claude Code which research output was the best and to my surprise it said Codex! I then asked it to add to the Codex research output any insights from other research outputs.</p> <p>Once we had compiled this information, we moved on to the next step -- architecting the solution.</p> <h1 id="architecture-plan">Architecture plan</h1> <p>The purpose of this phase is to commit to a concrete architecture/design and set of features based on the knowledge gathered from the previous phase. I asked Claude Code to create an architecture plan and then asked it to review it while brainstorming with Codex (you can set up the Codex MCP in your <code>.mcp.json</code> so Claude can use Codex). This created me a sensible design plan that looks roughly like so:</p> <pre class="giallo z-code"><code data-lang="markdown"><span class="giallo-l"><span class="z-markup z-heading">#</span><span class="z-entity z-name"> &lt;</span><span class="z-entity z-name z-tag">Feature</span><span class="z-entity z-name">&gt;</span><span class="z-entity z-name"> architecture plan</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Goal</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Context / problem</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Target architecture</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> User flow</span></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Storage model</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Safety constraints</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> metadata / validation context</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> permissions / limits / performance guardrails / failure handling</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Scope</span></span></code></pre><h1 id="implementation-plan">Implementation plan</h1> <p>The purpose of this phase is to commit to a specific implementation of the architecture we've specified above. Here again, I used the same process as before and asked Claude Code to draft an implementation plan and then asked it to review it with Codex. The implementation plan was defined as a series of steps but the plan was so huge that I ended up cutting it into multiple phases where each phase was its own plan:</p> <ul> <li>foundation with query compilation</li> <li>routing seam</li> <li>dashboard builder UI</li> <li>save, edit and delete flow</li> </ul> <pre class="giallo z-code"><code data-lang="markdown"><span class="giallo-l"><span class="z-markup z-heading">#</span><span class="z-entity z-name"> &lt;</span><span class="z-entity z-name z-tag">Feature</span><span class="z-entity z-name">&gt;</span><span class="z-entity z-name"> implementation plan</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Goal</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Execution policy</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> test-first / commit cadence / validation expectations</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Safety guardrails</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> invariants that must never be broken</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Completion gate</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> format / compile / lint / tests / QA required before done</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> File plan</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> New files</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> Modified files</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> Test files</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">##</span><span class="z-entity z-name"> Tasks</span></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> Task 1: </span><span class="z-entity z-name">&lt;</span><span class="z-entity z-name z-tag">name</span><span class="z-entity z-name">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Purpose</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Files</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Required outcomes</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Test outlines</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Manual spot checks</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> Step-by-step checklist</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-markup z-heading">###</span><span class="z-entity z-name"> Task 2: </span><span class="z-entity z-name">&lt;</span><span class="z-entity z-name z-tag">name</span><span class="z-entity z-name">&gt;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-list z-begin z-markdown">-</span><span> same shape</span></span> <span class="giallo-l"></span></code></pre><h1 id="implementation">Implementation</h1> <p>For the implementation phase I told Codex to:</p> <ul> <li>create a git worktree for the implementation phase</li> <li>follow the plan</li> <li>create a PR, assign it to me</li> <li>move on to the next phase of the plan</li> </ul> <p>Implementation was not a straight line. The backend-heavy parts responded well to planning and agent execution, but the UI exposed a gap in the spec. The first versions were technically functional but hard to understand and visually inconsistent with the rest of the app. At that point I stopped trying to prompt my way out of an underspecified UI problem and wrote down a proper design direction instead: first a design-system HTML file based on the existing app, then low-fidelity wireframes, then a higher-fidelity wireframe, and only then another implementation pass.</p> <p>So I changed the process:</p> <ul> <li>asked both Claude Code and Codex to produce a design system in a self-contained .html file out of what we already had in the application. Claude Code was much better than Codex here so this is the design system I selected.</li> <li>asked Claude via Claude.ai to implement a low fidelity wireframe (that we iterated over multiple times)</li> <li>asked Claude via Claude.ai to implement a high fidelity wireframe using the previously created design system</li> </ul> <p>I used Claude via Claude.ai instead of via Claude Code as I didn't want Claude to be distracted by existing code and practices, etc.</p> <h1 id="conclusion">Conclusion</h1> <p>Three things I'd do again:</p> <ul> <li>use multiple models early for research and planning, then keep the best output instead of forcing one model to do everything</li> <li>treat visual builders as two separate problems: query architecture and UI specification</li> <li>store structured authoring state for visual features, and only compile to SQL at the execution boundary</li> </ul> <p>That combination let me move fast without turning the feature into a future maintenance problem. I would never have gotten the same result in the pre-agentic coding era.</p> <p>The next thing I want to build is a more reactive development system around this workflow: when I move a PR from draft to ready for review, an LLM reviews it automatically; when CI goes red, an agent starts trying to make it green; when a design changes, downstream implementation work updates with it.</p> <p>That kind of system creates more parallel work so I can spend more time on product, architecture, and judgment instead of manually pushing every step forward.</p> <p>If you're building complex B2B interfaces with coding agents, I'd be interested to compare workflows—especially where the agents struggled with product or UI decisions. <a href="mailto:ryan.zidago@protonmail.com">Send me an email</a>.</p> 2026-03-08T00:00:02+00:00 2026-03-08T00:00:02+00:00 https://ryanzidago.com/posts/incrementally-enforcing-code-quality-rules-with-coding-agents/ <p>AI agents are becoming part of the development workflow, and they're only as good as the feedback your codebase gives them.</p> <p>OpenAI calls this <a rel="noopener nofollow noreferrer external" target="_blank" href="https://openai.com/index/harness-engineering/">harness engineering</a>: the idea that you should wrap your codebase in linters, structural tests, and automated checks that act as guardrails for both human developers and AI agents.</p> <blockquote> <p>In practice, we enforce these rules with custom linters and structural tests, plus a small set of "taste invariants." [...] we statically enforce structured logging, naming conventions for schemas and types, file size limits, and platform-specific reliability requirements with custom lints.</p> </blockquote> <p>Great. But what if your codebase doesn't have any of this?</p> <h2 id="the-problem">The problem</h2> <p>In a new project, it's fairly easy to add new linter rules or CI checks as there are very few existing offenses against those rules.</p> <p>In a large codebase that has seen many developers over the years, this is another story.</p> <p>You may have never used types or lints. You add them; now you get hundreds and hundreds of errors, lints, warnings, refactoring opportunities etc.</p> <p>How do you best proceed then? Do you let the AI fix everything, everywhere, all at once? You can end up with PRs with +10_000 lines of changes ... This is too much to review, so the probability that bugs slip through is higher. It's also a merge conflict magnet against any parallel work, and if something does break, rolling back means losing all the fixes; not just the problematic one.</p> <h2 id="the-strategy">The strategy</h2> <p>Here's what I suggest:</p> <ul> <li>add the tool</li> <li>enable it in the CI</li> <li>now locally, run the tool to see which config/rules produce offenses</li> <li>disable any rules that produce offenses and merge your PR into main</li> </ul> <p>This will prevent contributors from producing more offenses and set the best possible baseline.</p> <p>Now that you have a best case, let's look at how to bring the rest of the rules into play. Ask your favorite coding agent:</p> <blockquote> <p>Here's the list of rules that we want to enable in this project. Create stacked PRs where each PR enables (and eventually fix all offenses related to) one rule. Once all the tests and precommit checks and the CI passes, create the PR and move on to the next rule; rinse and repeat until all the following rules are enabled.</p> </blockquote> <p>A word of caution: not all rules are equal. Fixing import ordering is mechanical; adding types to a legacy module with implicit contracts can surface deeper design issues. If a rule's fixes start snowballing, it's fine to skip it and come back later. The point is steady progress.</p> <p>Here's what the progression looks like:</p> <table><thead><tr><th>Rule</th><th>PR #0</th><th>PR #1</th><th>PR #2</th><th>PR #3</th><th>PR #4</th><th>PR #5</th><th>PR #6</th></tr></thead><tbody> <tr><td><code>eqeqeq</code></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr> <tr><td><code>no-eval</code></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr> <tr><td><code>import/order</code></td><td>✕</td><td><strong>✓</strong></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr> <tr><td><code>no-console</code></td><td>✕</td><td>✕</td><td><strong>✓</strong></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr> <tr><td><code>no-unused-vars</code></td><td>✕</td><td>✕</td><td>✕</td><td><strong>✓</strong></td><td>✓</td><td>✓</td><td>✓</td></tr> <tr><td><code>naming-convention</code></td><td>✕</td><td>✕</td><td>✕</td><td>✕</td><td><strong>✓</strong></td><td>✓</td><td>✓</td></tr> <tr><td><code>no-explicit-any</code></td><td>✕</td><td>✕</td><td>✕</td><td>✕</td><td>✕</td><td><strong>✓</strong></td><td>✓</td></tr> <tr><td><code>strict-null-checks</code></td><td>✕</td><td>✕</td><td>✕</td><td>✕</td><td>✕</td><td>✕</td><td><strong>✓</strong></td></tr> </tbody></table> <p><strong>PR #0</strong>: Add tool to CI. Enable <code>eqeqeq</code> and <code>no-eval</code> (zero offenses). Disable the rest. <strong>PR #1–#6</strong>: One rule per PR. Fix all offenses. Merge. Next.</p> <p>There are various ways to slice this incremental work:</p> <ul> <li>per rule</li> <li>per rule per area (backend, frontend)</li> <li>per rule per domain (accounts, invoices, orders)</li> <li>per file</li> </ul> <p>Find the granularity where you are comfortable reviewing and releasing changes.</p> <p>The investment pays off twice: a cleaner codebase for your team and better guardrails for AI-assisted development.</p>