Performance Accelerators
This chapter documents the performance behaviours of Novomodelo’s SDDP solver that a user configures or observes: what to tune and what to read back from the output. Each accelerator addresses a specific cost driver in the training loop and is active by default unless noted otherwise. Understanding them helps users interpret timing statistics, configure cut management strategies, and diagnose performance regressions. The crate-internal data structures and allocation discipline that back these behaviours are documented in the novomodelo developer surface — see See Also.
Diagnosing Performance
Seção intitulada “Diagnosing Performance”The timing tables and the printed Time split hold rank 0’s view, so another
rank’s imbalance appears as rank 0’s time in cut_sync_ms, state_exchange_ms
and mpi_allreduce_ms. The backward wait is the exception: bwd_load_imbalance_ms
and the printed Backward wait cover the workers of every rank, so they include
the time a rank waits for slower peers to finish a stage’s solves. Reading the Time split breakdown
explains how to read the printed block. An enumerated backward pass records zero in
cut_sync_ms, state_exchange_ms and bwd_load_imbalance_ms.
| Symptom | Metric | Knob |
|---|---|---|
| Backward time grows each iteration | time_backward_ms and cuts_active in training/convergence.parquet; cuts_active_after and active_after_budget in training/cut_selection/iterations.parquet | training.cut_selection.selection and training.cut_selection.max_active_per_stage |
| Lazy-selection solves load many rows | mean_rows_in_lp in training/convergence.parquet (0 when no lazy selection ran); lazy_scoring_ms in training/timing/iterations.parquet | The dynamic method’s parameters under training.cut_selection.selection |
| Workers wait inside a phase | fwd_load_imbalance_ms and bwd_load_imbalance_ms in training/timing/iterations.parquet; the wait share of each phase in the printed time split | Backward: by_node with block_size under training.parallelism.backward_scheduler, see By-Node Scheduling (no effect while lazy selection is active). Forward: a training.selection.forward_passes that is a multiple of the worker count (ranks times --threads) |
| Workers idle for want of trajectories | forward_passes in training/convergence.parquet against distribution.world_size and distribution.threads_per_rank in training/metadata.json | Raise training.selection.forward_passes, or run fewer ranks or fewer threads per rank; see Parallel Execution Model |
| Collective time on multi-rank runs | cut_sync_ms, state_exchange_ms and mpi_allreduce_ms in training/timing/iterations.parquet | Balance the work as in the two rows above, or run fewer ranks with more --threads each, typically one rank per node; see Parallel Execution Model |
| Lower-bound evaluation takes a growing share of iteration time | lower_bound_ms in training/timing/iterations.parquet | None: the evaluation is serial on rank 0, one LP solve per stage-0 opening; see Lower Bound Evaluation |
| Solves retry | lp_retries and retry_attempts in training/solver/iterations.parquet; retry_level in training/solver/retry_histogram.parquet; the LP prescaling diagnostics in training/scaling_report.json | The per-phase solver profiles training.solver.backward, training.solver.forward and simulation.solver; modeling.cost_scale_factor |
| Balances recomputed from outputs do not close | lp_row_violations in training/solver/iterations.parquet and simulation/solver/iterations.parquet; the end-of-run warning of Solves that miss a row bound | None: report the case with the counts |
| Slow start | setup.load_seconds, setup.stochastic_fit_seconds, setup.production_fit_seconds, setup.evaporation_fit_seconds and setup.broadcast_seconds in training/metadata.json (collected on rank 0) | For the stochastic fit, export the fitted artifacts once and reuse them: exports.stochastic and Exporting Stochastic Artifacts |
Solver Safeguards
Seção intitulada “Solver Safeguards”When HiGHS returns a non-terminal error, the solver automatically escalates through a bounded retry sequence rather than surfacing the failure. The caller never sees intermediate failures — only the final solution or a single classified solver error. Escalation is capped by a built-in per-attempt iteration limit and per-level and overall wall-clock budgets, so a pathological solve cannot stall the run indefinitely.
How far escalation went on a given solve is recorded in the retry_level
output column (values 0–11) and written to
training/solver/retry_histogram.parquet for post-run analysis; a solve that
never retries adds no histogram row; level 0 is the first retry level. The escalation strategy itself — which options
each level applies — is a solver-internal detail documented in the
novomodelo-solver README.
LP Scaling
Seção intitulada “LP Scaling”Before each stage’s LP template is built, Novomodelo applies its own geometric-mean row/column prescaler that normalizes the constraint-matrix coefficients toward 1.0, improving numerical conditioning. The prescaling mechanism and its exact-unscaling guarantee are documented in the math layer under LP Layout and Scaling.
Cost Scale Factor
Seção intitulada “Cost Scale Factor”The objective is divided by a cost-scale factor (default 1_000_000.0,
configurable as modeling.cost_scale_factor)
to reduce the magnitude of objective coefficients, improving simplex numerical
stability.
Because the prescaler already conditions the matrix, HiGHS’s internal scaling
(simplex_scale_strategy) is disabled by default in every solver profile. It is
overridable per phase via the solver profile’s scale field: selecting
solver_scaling re-enables solver-side scaling and forgoes the exact
single-unscaling guarantee. See
Solver profile fields
for the field and its trade-off.
The scaling diagnostics are written to training/scaling_report.json after
template construction, documenting the coefficient range before and after
scaling for each stage.
Cut Management Pipeline
Seção intitulada “Cut Management Pipeline”As training progresses, the row pool grows and LP solve times increase. Novomodelo provides a two-stage row management pipeline (Stage 1, the selection pass, then Stage 2, the budget pass) to control this growth.
The pipeline runs after each iteration’s backward pass and cut synchronization:
Stage 1: Strategy-Based Selection
Seção intitulada “Stage 1: Strategy-Based Selection”Two methods are available, configured through
training.cut_selection in
config.json; the Configuration reference owns the keys and defaults:
| Strategy | Selection rule |
|---|---|
level1 | At each stored trial state, the cut with the highest value owns the state (an exact tie goes to the lowest pool slot); a scored cut is active exactly when it owns a state |
dynamic | Lazy row generation: each solve starts from a resident subset of the active cuts and grows it with the cuts the current LP solution violates; deactivates nothing |
level1 respects check_frequency: the selection pass runs only at iterations
that are multiples of it. It selects the pools of the stages strictly between
the first and the last; the first stage’s pool and the terminal pool are never
selected. Each such pool stores the trial state of every cut the run adds there,
or those of the last lookback_iterations iterations, and every scored cut,
current-iteration cuts and inactive cuts included, competes at every stored
state, so a deactivated cut is reactivated when it later owns a state. The rule
is in Level 1 Selection.
dynamic operates differently: it is a per-solve lazy loop that adds cuts on
demand rather than deactivating from a pool scan, and it does not read
check_frequency. See The Dynamic Method
for the procedure. A solve that reaches the round limit with a candidate still
violated is counted in lazy_round_cap_hits of
training/convergence.parquet.
Stage 2: Budget Enforcement
Seção intitulada “Stage 2: Budget Enforcement”A hard cap on LP size, enabled via max_active_per_stage. It applies to the
pools of the stages strictly between the first and the last, each node’s pool
separately, and counts only the cuts the current run generated: cuts loaded from
a policy, boundary cuts among them, neither count nor are evicted. When the count
exceeds the cap after the selection pass, the pool evicts cuts of earlier
iterations in ascending order of the iteration in which a cut last bound, then
of the number of solves in which it bound, then of its pool slot. Cuts of the
current iteration count but are never evicted. Under level1, a later
selection pass reactivates an evicted cut that owns a state, and the cap is
applied again in that iteration. The cap is refused with the dynamic method
and with an enumerated training.selection.
Unlike the selection pass, budget enforcement runs every iteration (not gated
by check_frequency).
Why it matters: High-parallelism configurations (many forward passes, few iterations) accumulate more active rows than low-parallelism configurations (fewer forward passes, more iterations), making each backward LP solve proportionally more expensive. Bounding LP size makes high-parallelism configurations viable without unbounded solve-time growth.
Observability
Seção intitulada “Observability”The row management pipeline writes per-pool statistics to
training/cut_selection/iterations.parquet,
where each column is defined. The knob table names the
columns that diagnose backward-time growth.
Parallel Execution Model
Seção intitulada “Parallel Execution Model”A run has two levels of parallelism: MPI ranks, and worker threads inside each
rank. --threads sets the number of worker threads per rank (default 1, at
least 1); the launcher sets the rank count (see
HPC & Cluster Deployment).
The Execution block of the run banner reports both levels. The excerpt below
shows an example two-rank run with two threads per rank on one host; it omits the
Solver line and writes the hostname as <host>:
Execution Backend: MPI (MPICH 4.3.2, MPI 4.1) Threads: Funneled, 2 rayon threads per rank Layout: 2 ranks on <host>The Threads: line gives the MPI thread level and the worker threads per rank;
the Layout: line gives the rank count and the host. A run on several hosts
shows across N nodes there, with the ranks of each host listed below it.
Work Partition
Seção intitulada “Work Partition”Under a sampled forward pass, the forward_passes trajectories of an iteration
(see training.selection) are
split over the ranks in contiguous blocks. Each rank takes
forward_passes / ranks of them, rounded down, and the first
forward_passes mod ranks ranks take one extra: 50 passes on 4 ranks give 13,
13, 12 and 12, and ranks beyond the number of passes receive none. A rank then
splits its block
statically over its worker threads the same way, so the assignment of
trajectories to threads depends on the counts only, never on thread
scheduling. Simulation scenarios are split over ranks and threads the same way.
Each rank runs the backward pass for the trial points of its own forward passes; the work units handed to its worker threads are defined under Backward-Pass Scheduling. Under an enumerated forward pass, the backward pass instead splits the successor outcomes of each node across the ranks, which allgather them so that every rank derives the same cut.
Every rank must run with the same --threads value: the backward pass of a
sampled forward pass compares the worker counts across the ranks and stops the
run when they differ.
Backward-Pass Scheduling
Seção intitulada “Backward-Pass Scheduling”By default (by_scenario), each parallel backward-pass work unit is one whole
trial point. The optional by_node scheduler uses finer
(trial point, opening block) work units for better load balance when trial
points solve in variable time. Either way the produced cut set and lower bound
are independent of worker and thread count. See
parallelism.backward_scheduler for the
configuration, and the
novomodelo-sddp README
for the scheduler and inter-rank communication internals.
By-Node Scheduling
Seção intitulada “By-Node Scheduling”The by_node scheduler shrinks the work unit from one trial point to one
(trial point, opening block) pair, giving the work-stealing counter more,
finer-grained units to hand out. This improves load balance specifically when
trial points solve in variable time: a worker that finishes a cheap trial
point quickly can pick up a block from a still-running expensive one instead
of idling until an entire trial point frees up.
Within a stage, opening-blocks are claimed hardest-first: blocks are
ranked by the previous iteration’s mean simplex pivot cost per (stage, block), descending, so the most expensive blocks are claimed early rather
than left for whichever worker happens to reach them last. This claim order
changes only which worker processes which block, and when — it is
result-neutral: the produced cut set and the training lower bound are
identical to canonical (whole-trial-point) claiming.
See parallelism.backward_scheduler
for the config field that enables by_node and sets its block_size.
Per-Phase Solver Profiles
Seção intitulada “Per-Phase Solver Profiles”Each algorithmic phase — forward sweep, backward sweep, and simulation — can
override the built-in solver profile (feasibility tolerances, the simplex
update limit, pricing strategy, the scale field, and more) via
training.solver.backward, training.solver.forward, and simulation.solver.
See Solver profile fields
for the full field list, the built-in profile each phase starts from, and the
single-unscaling trade-off carried by scale.
Overriding the backward phase with tighter feasibility tolerances can reduce backward-pass solve-time variance, which improves load balance across worker threads and shortens wall-clock training time.
Forward Pass and Simulation
Seção intitulada “Forward Pass and Simulation”Under a sampled forward pass and in simulation, scenarios are assigned to workers by the static partition described under Work Partition. Within each scenario the stage LP is loaded once. Per stage, the forward-pass and simulation solves patch the incoming-state column bounds, the stochastic NCS generation column bounds, and the load-balance and inflow row bounds.
Lower Bound Evaluation
Seção intitulada “Lower Bound Evaluation”The lower-bound evaluation (solving a stage-0 LP for every opening in the tree) runs as a single-threaded serial loop on rank 0. Each opening patches correctness-critical per-opening state (e.g. NCS column bounds) on a shared solver in sequence, so the step is not parallelized.
Communication and Threads
Seção intitulada “Communication and Threads”The ranks exchange data through blocking MPI collectives, so communication does
not overlap computation. MPI is initialized at the Funneled thread level,
which the Threads: line of the Execution block names: only one thread of a
rank makes MPI calls. An allgatherv gives every rank the data of all ranks, an
allreduce combines a value over all ranks, and a broadcast sends a value from
rank 0 to the others. On every run the main collectives are:
- an allgatherv of the forward-pass costs, reduced in canonical order, so the result does not depend on the rank count;
- allreduce sums of integer counters, among them the solve counts of the printed summary;
- a broadcast of the lower bound, which rank 0 evaluates (see Lower Bound Evaluation) and the other ranks wait for.
Under a sampled forward pass the backward pass adds:
- an allgatherv of the trial states of all ranks at the start of each backward
stage (timed as
state_exchange_ms); - a per-stage allgatherv of the cuts generated in the backward pass, at the end
of each backward stage, which every rank waits on (timed as
cut_sync_ms); - an allreduce sum of the per-iteration cut-binding increments.
The enumerated backward pass has neither the state exchange nor the cut
exchange; it uses an allgatherv of the successor outcomes of each node instead
(see Work Partition). The communicator backends are documented in
the novomodelo-comm README.
Each LP solve runs on one thread: the bundled HiGHS is built without its
default threads, so during the forward pass, the sampled backward pass and
simulation the number of LPs a rank solves at the same time is its --threads.
The enumerated backward pass runs on one worker thread per rank.
Rank 0 writes the training outputs. The timing tables and the printed
Time split show rank 0’s view: the phase wall, forward wait, and serial values
are rank 0’s own, not an aggregate over the ranks. The backward wait
(bwd_load_imbalance_ms) is taken over the workers of every rank. The printed
solve counts are sums over all ranks, and the printed solve time is the mean
per worker over all ranks.
Memory and Case Access
Seção intitulada “Memory and Case Access”Each rank holds its own full copy of the case data, the opening tree, the cut
pool, and the stage templates; ranks on one node do not share memory. The cut
pool is sized at startup from the iteration budget (the largest iteration_limit
in the stopping rules) times the forward passes, for
every stage. The memory of every rank is set by that size, and placing more ranks
on a node multiplies that node’s memory use.
Rank 0 loads the case and broadcasts the system data, the configuration, and any user-supplied opening tree; the other ranks rebuild the stochastic context and the hydro models from the case directory. Every rank must read the case directory, so it must be visible at the same path on every node (a shared filesystem or a staged copy). A resume or warm start also loads the policy checkpoint on every rank, from the output directory.
See Also
Seção intitulada “See Also”- Configuration — row-selection and row management configuration
- Output Format — timing, solver statistics, and row-selection output schemas
- novomodelo-solver — solver interface and retry-escalation internals (novomodelo developer guide)
- novomodelo-sddp — training-loop architecture, scheduling, and data structures (novomodelo developer guide)
- novomodelo-comm — communicator backends and collectives (novomodelo developer guide)
- ARCHITECTURE.md — novomodelo crate topology and dependency graph (novomodelo developer guide)