Using doctests to understand a feature

TLDR:

• Ask for an explanation.
• Turn uncertain claims into executable examples.
• Run them against the existing implementation.
• Correct the explanation from the results.
• Keep the useful examples beside the code

I've already written extensively about:

• the importance of executable evidence
• doctests for documenting known limitations and high level product processes

The more I work with LLMs, the more I realise how powerful executable documentation with doctests is. So here's another article about them:

LLMs allow me to be drastically more ambitious and iterate much faster. Everyone around me is doing the same as well.

As such, when implementation complexity and scope increase, using doctests to specify the current behaviour of the system is helpful to anchor LLMs' answers in verifiable statements and make them readable to humans.

Lately at work, I needed to quickly understand a core module that we use and that depends on many other features. So I started by asking the LLM to provide me with the State Of The Art about that feature (this is the documentation part). It returned tons of text. I wasn't certain of the veracity of some statements, so I asked it to generate some tests for me (with LLMs, I use tests more and more as a tool to understand the current behaviour of the system - they are a tool in themselves - may be worth an article later on). Then it corrected some of its statements. I thought it was pretty cool (this is the test part).

But then I realised that I had docs and tests ... this is actually what doctests are for!

This is a rough fictional moduledoc with doctests for a procurement platform - it follows the following pattern

## Section
What (observed behaviour)
Why (product rationale)
<concrete examples with doctests>

( Side note: the why is super important and often underrated ... it is the thing that will tell you some time in the future whether the assumption embedded into the Why still holds true or not !! So be kind to your future self and AI agents AND MENTION WHY!!! )

@moduledoc """
Plans purchase order cancellations across supplier fulfilment, purchasing
permissions, departmental budgets, payments and accounting.

## Purchasing authority

Only procurement managers from the purchasing organisation may request
cancellation. Cancelling an order changes the company's commitments to
suppliers, so ordinary purchasing access is insufficient.

    iex> order = %Procurement.PurchaseOrder{organisation_id: 42}
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42,
    ...>   role: :requester
    ...> }
    iex> Procurement.Cancellations.plan(order, membership)
    {:error, :not_authorised}

    iex> order = %Procurement.PurchaseOrder{organisation_id: 42}
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 99,
    ...>   role: :procurement_manager
    ...> }
    iex> Procurement.Cancellations.plan(order, membership)
    {:error, :not_authorised}

## Partial fulfilment

Only quantities that have neither shipped nor already been cancelled
can be cancelled. Shipped goods require a return, while previously
cancelled quantities must not generate another financial adjustment.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101,
    ...>       quantity: 10,
    ...>       shipped_quantity: 4,
    ...>       cancelled_quantity: 2,
    ...>       unit_price_cents: 30000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership)
    iex> Enum.map(plan.lines, &{&1.purchase_order_line_id, &1.quantity})
    [{101, 4}]
    iex> Procurement.Cancellations.plan(order, membership, quantities: %{101 => 5})
    {:error, {:quantity_exceeds_cancellable, 101, 4}}

## Supplier cancellation deadlines

Cancellation must be requested before the deadline agreed with the
supplier. After that point, the supplier may already have committed
production capacity or purchased materials, even if nothing has shipped.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   cancellation_deadline: ~U[2026-10-07 12:00:00Z],
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 1, unit_price_cents: 30000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> Procurement.Cancellations.plan(order, membership,
    ...>   at: ~U[2026-10-07 12:00:00Z]
    ...> )
    {:error, :cancellation_deadline_passed}

## Made-to-order goods

Made-to-order items require explicit supplier approval before cancellation.
These goods may have little resale value, so being unshipped does not
mean the supplier can recover its costs.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101,
    ...>       quantity: 20,
    ...>       unit_price_cents: 15000,
    ...>       cancellation_policy: :supplier_approval
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> Procurement.Cancellations.plan(order, membership)
    {:error, {:supplier_approval_required, [101]}}

## Departmental budget allocation

Released commitments return to the budgets that funded the cancelled
quantities. A purchase order may serve several departments, and returning
everything to one budget would distort their remaining spending capacity.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   payment_status: :unpaid,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, budget_id: 17, quantity: 2, unit_price_cents: 30000
    ...>     },
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 102, budget_id: 18, quantity: 3, unit_price_cents: 20000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership,
    ...>   quantities: %{101 => 1, 102 => 3}
    ...> )
    iex> plan.budget_releases
    ...> |> Enum.map(&{&1.budget_id, &1.amount_cents})
    ...> |> Enum.sort()
    [{17, 30000}, {18, 60000}]

## Unpaid purchases

Cancelling an unpaid purchase releases its budget commitment without
issuing a cash refund. A commitment records intended spending; refunding
it would transfer money that was never collected.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   payment_status: :unpaid,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, budget_id: 17, quantity: 2, unit_price_cents: 30000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership)
    iex> Enum.sum(Enum.map(plan.budget_releases, & &1.amount_cents))
    60000
    iex> plan.refund_cents
    0

## Cancellation fees

For prepaid purchases, the supplier's agreed cancellation fee is deducted
from the refund. The fee remains an expense because cancellation does
not erase costs the buyer has contractually agreed to cover.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   payment_status: :paid,
    ...>   cancellation_fee_basis_points: 1000,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 2, unit_price_cents: 30000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership)
    iex> plan.cancellation_fee_cents
    6000
    iex> plan.refund_cents
    54000

## Issued invoices and tax

Invoiced quantities require a credit note using the original invoice's
amounts and tax treatment. Cancelling the order alone would leave the
invoice payable, while using current prices or tax rates could produce
an adjustment that does not reconcile with the original charge.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   payment_status: :unpaid,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 2, unit_price_cents: 30000
    ...>     }
    ...>   ],
    ...>   invoices: [
    ...>     %Procurement.Invoice{
    ...>       id: 501,
    ...>       status: :issued,
    ...>       lines: [
    ...>         %Procurement.InvoiceLine{
    ...>           purchase_order_line_id: 101,
    ...>           quantity: 2,
    ...>           net_cents: 60000,
    ...>           tax_cents: 12000
    ...>         }
    ...>       ]
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership,
    ...>   quantities: %{101 => 1}
    ...> )
    iex> Enum.map(plan.credit_notes, &{&1.invoice_id, &1.amount_cents})
    [{501, 36000}]
    iex> plan.refund_cents
    0

## Quantity-dependent discounts

When supplier terms require repricing, cancellation can remove a volume
discount from retained quantities. The refund must include that adjustment
because the original price depended on purchasing the qualifying quantity.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   payment_status: :paid,
    ...>   reprice_on_cancellation: true,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101,
    ...>       quantity: 10,
    ...>       unit_price_cents: 9000,
    ...>       price_breaks: [
    ...>         %Procurement.PriceBreak{
    ...>           minimum_quantity: 1, unit_price_cents: 10000
    ...>         },
    ...>         %Procurement.PriceBreak{
    ...>           minimum_quantity: 10, unit_price_cents: 9000
    ...>         }
    ...>       ]
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership,
    ...>   quantities: %{101 => 2}
    ...> )
    iex> plan.cancelled_value_cents
    18000
    iex> plan.retained_items_price_increase_cents
    8000
    iex> plan.refund_cents
    10000

## Supplier minimum order value

Partial cancellation cannot leave an order below the supplier's agreed
minimum value. The supplier accepted the purchase on terms that make
fulfilment economical; reducing it below that minimum requires renegotiation.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   minimum_retained_value_cents: 50000,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 4, unit_price_cents: 20000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> Procurement.Cancellations.plan(order, membership,
    ...>   quantities: %{101 => 3}
    ...> )
    {:error, {:below_supplier_minimum, 20000, 50000}}

## Financial approval limits

Cancellations above a manager's delegated limit require finance approval.
Large reversals can materially change cash forecasts and purchasing
commitments, even when the manager can normally cancel orders.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 20, unit_price_cents: 30000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42,
    ...>   role: :procurement_manager,
    ...>   cancellation_limit_cents: 500000,
    ...>   cancellation_limit_currency: "EUR"
    ...> }
    iex> Procurement.Cancellations.plan(order, membership)
    {:error, {:finance_approval_required, 600000}}

## Purchase currency

Refunds retain the purchase currency, even when the organisation reports
in another currency. Currency conversion belongs to settlement and
reporting; applying today's exchange rate here would change the amount
owed under the original purchase.

    iex> order = %Procurement.PurchaseOrder{
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "GBP",
    ...>   reporting_currency: "EUR",
    ...>   payment_status: :paid,
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 2, unit_price_cents: 25000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   organisation_id: 42, role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership)
    iex> {plan.currency, plan.refund_cents}
    {"GBP", 50000}

## Audit history

The plan includes an audit event identifying the requester, affected order
and supplied reason. It is persisted when cancellation is applied so that
finance and procurement can explain the change without reconstructing it
from emails or recording an unapplied plan as a completed cancellation.

    iex> order = %Procurement.PurchaseOrder{
    ...>   id: 1001,
    ...>   organisation_id: 42,
    ...>   status: :confirmed,
    ...>   currency: "EUR",
    ...>   lines: [
    ...>     %Procurement.PurchaseOrderLine{
    ...>       id: 101, quantity: 1, unit_price_cents: 30000
    ...>     }
    ...>   ]
    ...> }
    iex> membership = %Procurement.Membership{
    ...>   user_id: 23,
    ...>   organisation_id: 42,
    ...>   role: :procurement_manager
    ...> }
    iex> {:ok, plan} = Procurement.Cancellations.plan(order, membership,
    ...>   reason: "Office opening postponed"
    ...> )
    iex> plan.audit_event.actor_id
    23
    iex> plan.audit_event.purchase_order_id
    1001
    iex> plan.audit_event.reason
    "Office opening postponed"
"""

You can now clearly see that the purpose of this executable spec is not to specify every possible observable behaviour but rather, to help humans gain a high level understanding of a particular feature.

Of course it has limitations. One example limitation is that, irrespective of the size of the moduledoc above, there are absolutely zero guarantees that all relevant high level observable behaviours are actually encoded into the spec. Still, it is, at least for me, a marginally better problem to have than dealing with prose and tests separately.

What I love about this:

1. it reduces incorrect answers
2. certain statements are easier to understand in code than in English
3. having a high-level overview of what a feature does helps me make better decisions

In my opinion, the ability to understand extremely fast how a current feature behaves and why will be more and more important in the future, as we detach ourselves from implementation by delegating to AI agents and AI agents are able to ship code much faster than ever before.

Have fun with doctests!

I'm currently exploring the idea of collaborative documents with higherlevel.to to help product teams:

• define what correct looks like
• delegate to AI
• demand evidence It's in early beta and looking for feedback.