Roadmap

What we build toward and what we have decided never to build. Ordered work is in GitHub issues; this is the index and the rationale. Everything is judged against the ceiling in docs/design/ceiling.md: affine ∩ relational ∩ local.

How we measure

Not "can we say what other tools say?" — that has no finite answer — but "what is sayable in a language that must stay streamable?" A primitive is done when lowering.py accepts it and the differential test passes.

What we publish is coverage against two corpora, generated by a test rather than by hand (#27): Calliope's base math plus its published custom-math library, and PyPSA's optimization/constraints.py. Every item scores expressible, out of scope by design (data prep — SPEC §11), or genuinely arbitrary Python (an escape island, or stays a callback). No numbers are published until #27 produces them.

Expressiveness is not the axis we lead on. Cost is, on the one claim a reader can check rather than take: end-to-end cost to a loaded solver, wall and peak, against the eager lane's own best path to the same place (benchmarks). Build memory alone is not that claim — it is a small fraction of what solving costs, so shrinking it further changes nothing a caller feels. Two claims we would rather lead on are not measured yet and are therefore not made: the floor under the LP-file route as a cold cost, and marginal cost per model in a loop. Both are listed in Not measured yet; neither is quoted until a table backs it.

The degree axis

Of the three admissibility rules, degree is the only scope choice; relational and local fall out of the architecture. The plan is affine-by-design, quadratic out of scope — a decision with a revisit condition, not an axiom. It holds because demand is almost entirely quadratic objectives, and for convex 1-D curves we ship the substitutes practitioners prefer (piecewise: with convex: true, and the epigraph pattern in plain YAML) which keep LP duals, warm starts and MILP compatibility that a quadratic objective gives up. Quadratic constraints are a class HiGHS cannot solve at all, and every sink we ship assumes affine rows.

What would have to hold to move it. The old list said "sinks carry a quadratic stream", and that condition is already met — HiGHS passHessian and Gurobi setMObjective both exist, and linopy's QuadraticExpression means the oracle covers it. What remains is one blocker and one cost:

1. MIQP is excluded on the default solver — the blocker. HiGHS returns kError for Hessian + integrality, and binary:/integer: and nonconvex piecewise: are already in the language, so quadratic would conflict with shipped features. Gurobi has no such exclusion, which makes this a capability question: quadratic is the motivating case for Track 4's conjunction-exclusion axis, not an exception to it.
2. The Hessian is passed whole — a cost, not a rule-4 violation, since under the aligned-only scope Q is diagonal and small beside the solver's own model. The arithmetic and its two caveats are in docs/benchmarks.md.

Plus a data-time convexity guard, a new plan node, a lowering case and a label-ordering change — a bigger diff than SOS, which is a declaration and needs none of it. The question that actually decides it is whether the demand is LP-only: if the fuel-curve users also want unit-commitment binaries, (1) kills it on the default path however cheap the plumbing is, and piecewise: {convex: true} stays the answer.

An escape cannot lift degree either way: islands return affine COO rows. Scope if it lands, fixed in advance: degree ≤ 2, factors coordinate-aligned, objective only, convex-guarded — aligned products are a pointwise self-join; general bilinear coupling is a cross join with |terms|² rows and is excluded permanently.

Sequencing, not a verdict. The cost side is small and the oracle is free — linopy's QuadraticExpression builds the comparison, which is normally the expensive half of a new primitive. What is unsettled is not whether quadratic is worth having but where it can land: HiGHS refuses Hessian + integrality, and binary:, integer: and nonconvex piecewise: all ship today, so on the default path quadratic conflicts with features already in the language. That makes two things prerequisites rather than quadratic work — Track 4 declaring the conjunction exclusion, and a second solver bound to solver_direct that has no such exclusion (Gurobi, #106). Landing the primitive first ships math the default solver refuses.

The live argument — in particular whether coordinate-aligned is the right restriction, or whether the real axis is is the coupling declared as data? — is in #261 and #84, which supersede the two paragraphs above. This section states the sequencing; those state the math.

Track 1 — language primitives (taxed)

#	Primitive	Proposed surface	Unlocks	Locality	Status
1	Indexed access, constant	at(x, over=dim, index=value)	boundary conditions, multi-period linking, carrier ratios	pointwise	planned
2	Indexed access, lookup	at(x, over=dim, index=map)	DC power flow, hydro cascades — topology math beyond nodal balance	pointwise	planned
3	Coordinate helpers in where	index(dim, i), negative i from the end	"first/last snapshot" without hardcoding a label	coordinate-space	blocks 4
4	Window sum	sum_next_n(x, over=dim, n=N)	rolling horizons, DSR, reserves, min up/down time	bounded-halo	needs 3
5	Component-aware where	defined(v) over variables	masks compose across components instead of being re-derived and drifting	pointwise	needs design
6	all-reduction in where	all(x, over=dim)	reducing a mask over an out-of-foreach dim at all — today that is a DimensionError, since an implicit reduction would silently widen the mask	pointwise	small
7	Bounds as expressions	—	signs, bidirectional flows, component libraries	pointwise	#31
8	Namespacing / qualified names	—	component libraries	n/a	#29
9	Compose-then-build (schema merge)	—	one lower/stream pass over merged libraries	n/a	#30
10	Cumulative sum, data only	cumsum(p, over=dim) over a variable-free expression	investment schedules, cumulative budgets and caps — today a data-prep round trip for one column	coordinate-space	small
11	Semi-continuous variables	semi_continuous: true beside binary/integer	"off, or between min-stable and max" without the big-M pair it is written as today	n/a	small

Items 1–4 are one piece of work — indexed access plus the masks that make it usable — and are the difference between "dispatch and balances" and "network and multi-period math". Item 2 is the adjoint of group_sum: same mapping table, join without the aggregate. Calliope spells 1 and 2 as x[dim=value]; we do not, because a dimension in a kwarg key cannot be parameterised by a macro.

The window family is three questions, not one. They are worth separating because only the third is expensive, and the row in Deliberate non-primitives used to price all three as if they were the third:

• diff is x - shift(x, over=d, by=1). It needs nothing — it is a macro today, and edge= already answers what happens at the boundary.
• Fixed-width windows (item 4) are bounded-halo, the same locality class as shift, whose cyclic form already ships. Admissible, and gated only on item 3.
• cumsum splits on what it reduces. Over a variable-free expression it is one data column with no term expansion — free, and item 10 above. Over a variable the window is t rather than a constant, so row t carries t terms: an 8760-snapshot horizon emits ~38M nonzeros where the state-variable recurrence emits ~26k.

That last number is the whole argument, and it is not a build cost. Our own throughput is ours to fix and must never gate what the language can say. But a construct that inflates the emitted model is paid by the solver, on every solve, and no engineering on this side reduces it later. So cumsum over a variable is not refused on locality grounds — it is priced, and the open question is the mechanism: the label budget of #38 is a cap on rows and columns declared before the work runs, which is exactly the shape needed here, and generalising it from escape islands to any superlinear construct would make the size visible at load time rather than discovered in the solver. Deciding that also decides how much job is left for an island at all.

Item 11 is the cheapest thing on this table and the only one blocked by nothing: HiGHS carries kSemiContinuous and kSemiInteger natively, so unlike SOS (#23) and quadratic it needs no Track 4 finding on the default sink; linopy has Model.semi_continuous, so the oracle already covers it; and it adds a declaration flag, not a plan node — no expression grammar, no lowering case beyond the column type. The LP-file sink needs its own bound section, which is where the work actually is.

Tracks 2–3 — untaxed

Nothing here touches the build path or either lane's semantics, with one flagged exception noted under 2b.

• 2 — post-solve stage. Declared derived results (LCOE, curtailment, emissions by group) are SQL over solution tables we already materialise. Sequenced after Track 1 items 1–4.
• 2b — the operational surface. Mostly queries, and the work that decides whether the engine is usable at 3am: algebraic rendering of a row, IIS read-back where a solver provides one, model statistics and coefficient-range diagnostics. Dual read-back has shipped — sol.dual(constraint), the same label join as primal against the row table, refusing rather than zero-filling where a model has no dual solution; reduced costs and slacks ride the same join and have not (#78). The headline item is elastic relaxation, the infeasibility answer — HiGHS has no IIS, so solver_direct returns a status code for a model too large to open in an editor; slack variables with penalty costs, then a query of nonzero slacks grouped by constraint block, needs no solver feature, works on every sink, and points at which constraints conflict rather than at a minimal set. It is the exception on this track: adding slacks means new variables, so it needs a schema-level expansion pass the way piecewise: does, and is taxed like a primitive rather than free (#80).

The REPL gap is verbs, not structure. Anyone arriving from linopy loses the ability to hold a model in a session and poke at it, and that reads as the larger loss — but the substrate here is better, not worse: a built model is four polars frames, which answers "what is in row 12" with a filter, where a labelled Dataset answers it with a scatter. What is missing is that nothing on the built object reads them back. lps.build() already hands back the live executor; it just has nothing readable on it. Every verb worth having is a query over frames that already exist — render a bound row with its labels and coefficients (typeset/ renders the declaration, which is the other half), preview which rows survive a where before solving, evaluate an expression against bound data, coefficient ranges and model statistics. All of it is read-only: it inspects a built model and never declares one, which is the line that keeps this from becoming the Python modeling API rule 6 refuses. That line is what makes the whole block cheap, and it should be stated in the first issue rather than defended in each one. - 2c — value-only re-solve. In scope, because var_label is the solver column index with no remapping: changing a bound or an RHS is a label query plus changeColsBounds/changeRowsBounds, which is what rolling horizon, sweeps and Benders need. With one precondition: "value-only" has to be a property of the model, not of the user's intent. var_label is a ROW_NUMBER() over the rows that survive the where mask, so a parameter read by a where changes which rows exist and every later label shifts — an in-place bound update then writes to the wrong columns. A changed group_sum mapping leaves the labels alone but rewrites A, which a bounds-only update cannot express at all. Both are decidable off the resolved AST, and that check ships with the feature (#82). Structural editing stays out — it invalidates the label contract. Related gap: warm starts, which solver_direct would need to set through highspy.

Integrality is label-preserving too, and rides the same contract for free. Relaxing a MILP is one call in linopy (Variable.relax()); here it is a second YAML file or an edit, which is why "is my model even feasible as an LP" is harder than it should be. But var_label is a ROW_NUMBER() over the rows surviving the where mask, and integrality does not decide which rows exist — so changeColsIntegrality is exactly as safe as changeColsBounds, with none of the caveats above about where-read parameters or a changed group_sum — and the direct sink already makes that call to set integrality at build time, so the work is a flag on the session rather than a new path. It belongs on the solver session (#204) as a sibling of value rebinding, not in the language: relax-and-fix, rounding heuristics and LP bounds on a MILP are all how you solve a model, not what the model says. - 2d — decomposition, and why the shape favours us. Benders and successive substitution are the case 2c is for, and the architecture is better suited to them than an array-based layer is — worth stating, because the opposite is the natural assumption. Decomposition wants the model as sparse triplets plus label tables. That is not something we produce for the solver; it is the model, so every iteration reads what we already hold rather than re-deriving it. Each step of the algorithm is then a query rather than a scatter: the master/sub split is a GROUP BY over the A frame (#39 is exactly that analysis); cut coefficients are duals ⋈ subproblem rows, and dual read-back has shipped; and because var_label is the solver column index with no remapping, a cut in master variables lands with zero translation. foreach dims supply the block boundaries for free, which is the same concept #123 names for scenarios and pathways.

What is missing is small and already tracked: the session that holds the handle (#204), an addRows on it — the call the direct sink already makes to feed the initial build, so appending a cut is the same code path, not a new one — and #39. Appending costs nothing structurally: adding rows moves no column label and renumbers no existing row, so primal and dual keep working across cuts. Removal and reindexing are what the label contract refuses, and decomposition needs neither. The open question is not feasibility but who writes the cut, since rule 6 refuses a Python modeling API — a shipped driver reading the frames, or the blessed seam that composition (the ceiling) already forces. Same question, second direction; worth deciding once. - 3 — AST consumers (#21): math → LaTeX (a tree walk, no data — the product is the declarative math, so it should render as typeset documentation), CLI check/solve/write (#35), observability (#34).

Track 4 — sink capabilities

Decision: the ceiling and sink capability are two axes, not one. The closure (affine ∩ relational ∩ local) is about streamability and is solver-independent; what a sink can ingest is separate. Treating them as one let HiGHS's limits read as architectural law — the measured table is in docs/benchmarks.md.

Adopted: a declared capability per sink (not per solver — lp_file is not a solver but has capabilities), modelled on linopy's SolverFeature / Solver.features, with two divergences. Entries are three-valued (native / reformulated / absent) so satisfying a capability by reformulation later is additive; and the model expresses conjunction exclusions, because linopy declares HiGHS with INTEGER_VARIABLES and QUADRATIC_OBJECTIVE in one flat frozenset while HiGHS refuses the pair. check(model, sink=...) takes the sink optionally: bare check asks whether the model is sayable and lowerable, the argument asks whether that sink will take it. Quadratic is in scope as the case that forces the exclusion axis; SOS goes first on effort, not principle.

Hard rule 3 is untouched — it governs lanes accepting the same language, and naming an alternative sink is not the lane redirection it forbids.

Full design, work breakdown and open questions: #89. First spike is SOS2 on lp_file: no new dependency, no license question, no memory risk.

Track 5 — the memory axis

The engine holds the model it builds, so peak tracks the model rather than a number the caller sets. That is the right default — it is also what makes the lifetime disappear from the API — but it leaves two things open, and they are worth stating rather than discovering:

• A declared ceiling. There is no way today to say "build this within N gigabytes or fail". The honest version is not a knob bolted onto the engine but partition-wise execution: build the model in slices of the leading foreach dim and sink each slice, which the locality closure (pointwise ∩ bounded-halo) already guarantees is safe. Wanted when a model is written once and run on a machine chosen by someone else.
• The solver is the larger term anyway. Measured at roughly an order of magnitude above the build at 10⁷ variables (benchmarks), so a ceiling on the build is worth having for the write path — lps.write to LP or MPS, handed to something else — and worth much less when the same process goes on to solve.

Deliberate non-primitives

Billed vs banned: an escape: island (#38) returns affine COO rows, so it buys back what the relational and local rules exclude, priced against a label budget enforced before any Python runs. It cannot buy back degree. Reach for one only after #38's decision tree: different values → a parameter; different in-language shape → a second constraint with where; only unsayable shapes escape.

Request	Why not	Instead	Escape?
Quadratic / bilinear terms	degree pinned at 1	piecewise: (convex: true) or the epigraph pattern	banned today, blocked by the QP∧MILP exclusion on the default solver — a Track 4 capability question, and the bigger diff of the two (new plan node, lowering, convexity guard)
SOS / indicator (#23)	the default solver has no such concept	piecewise: λ-formulation (SOS2's usual purpose); big-M for indicator, given bounds	banned today, and the easier one: lp_file carries it as a text section and Gurobi natively — it needs Track 4, not a new invariant
Running sum over a variable	not locality — the emitted model is O(T²) rows	state-variable recurrence (a storage SOC balance is the rewrite)	priced, not banned — see the window family
Normalisations (x / sum(x))	degree, not locality — a variable divisor is refused by law 9	fix the denominator as a parameter, or state the ratio as a constraint	billed
Conditionals, iteration, data-dependent structure	destroys the closed AST	where masks + foreach dims; computation → data prep	billed — inside an island only
Resampling, clustering, IO, units	data layer, not math	preprocess; pass a parameter	n/a — free in Python already
Domain helpers (reduce_carrier_dim)	encodes one domain into the language	component libraries over generic primitives	n/a
Arbitrary array ops (merge, reindex)	unbounded; xarray with extra steps	data prep	billed

Honest snapshot

Cheaper here, because the model is tables: model statistics and coefficient-range diagnostics; IIS read-back (a join, not a scatter — as dual read-back, now shipped, already was); serialization (parquet, more portable than netCDF); elastic relaxation; and dualization — transposing a COO matrix is swapping two column names.

Ahead of comparable declarative layers: sparse-by-construction build with no dense intermediate, and a hand-off straight to the solver rather than through a file (see benchmarks; a declared memory ceiling is Track 5, not something we have); parameterised macros: (Calliope's sub-expressions take no arguments); binary and integer variables; piecewise as N links with per-link signs, convex mode and active gating; non-cyclic shift; load-time validation of every expression, where string and uncalled macro template.

Behind: Track 1 items 1–6 and 10–11, plus Tracks 2 and 3. Where linopy stays ahead it is not expressiveness but the post-solve object (labelled DataArrays vs our tidy tables — to_dataarray bridges), debugging (IIS via Gurobi, print() of a row, against our status code — until Track 2b exists), lifecycle (mutate, re-solve, warm start, relax/fix, callbacks vs our build-and-solve), solver breadth (ten backends and four handoffs vs HiGHS-direct plus LP files, #106), and one variable type we simply do not have (semi_continuous, item 11). None are ceiling questions; all are cheaper here once written, which is why they need scheduling rather than assuming they follow from the architecture.

The ranking that follows from this page, if it is read as a whole: indexed access (items 1–4) blocks whole model classes today; the Track 2b verbs block using the engine at 3am; solver breadth blocks arrival from linopy at all; and items 10–11 are cheap, unblocked and unscheduled.

Solvers we actually speak: HiGHS through solver_direct, Gurobi planned on the same path, LP files for everything else. Stated because anyone arriving from linopy will assume solver-agnosticism; the LP route costs the float→text→parse round trip solver_direct exists to remove, so "fast" is per-solver binding work rather than a property of the engine.