Skip to content

Performance Accelerators

This chapter documents the performance behaviours of Novomodelo’s SDDP solver that a user configures or observes: what to tune and what to read back from the output. Each accelerator addresses a specific cost driver in the training loop and is active by default unless noted otherwise. Understanding them helps users interpret timing statistics, configure cut management strategies, and diagnose performance regressions. The crate-internal data structures and allocation discipline that back these behaviours are documented in the novomodelo developer surface — see See Also.


The timing tables and the printed Time split hold rank 0’s view, so another rank’s imbalance appears as rank 0’s time in cut_sync_ms, state_exchange_ms and mpi_allreduce_ms. The backward wait is the exception: bwd_load_imbalance_ms and the printed Backward wait cover the workers of every rank, so they include the time a rank waits for slower peers to finish a stage’s solves. Reading the Time split breakdown explains how to read the printed block. An enumerated backward pass records zero in cut_sync_ms, state_exchange_ms and bwd_load_imbalance_ms.

SymptomMetricKnob
Backward time grows each iterationtime_backward_ms and cuts_active in training/convergence.parquet; cuts_active_after and active_after_budget in training/cut_selection/iterations.parquettraining.cut_selection.selection and training.cut_selection.max_active_per_stage
Lazy-selection solves load many rowsmean_rows_in_lp in training/convergence.parquet (0 when no lazy selection ran); lazy_scoring_ms in training/timing/iterations.parquetThe dynamic method’s parameters under training.cut_selection.selection
Workers wait inside a phasefwd_load_imbalance_ms and bwd_load_imbalance_ms in training/timing/iterations.parquet; the wait share of each phase in the printed time splitBackward: by_node with block_size under training.parallelism.backward_scheduler, see By-Node Scheduling (no effect while lazy selection is active). Forward: a training.selection.forward_passes that is a multiple of the worker count (ranks times --threads)
Workers idle for want of trajectoriesforward_passes in training/convergence.parquet against distribution.world_size and distribution.threads_per_rank in training/metadata.jsonRaise training.selection.forward_passes, or run fewer ranks or fewer threads per rank; see Parallel Execution Model
Collective time on multi-rank runscut_sync_ms, state_exchange_ms and mpi_allreduce_ms in training/timing/iterations.parquetBalance the work as in the two rows above, or run fewer ranks with more --threads each, typically one rank per node; see Parallel Execution Model
Lower-bound evaluation takes a growing share of iteration timelower_bound_ms in training/timing/iterations.parquetNone: the evaluation is serial on rank 0, one LP solve per stage-0 opening; see Lower Bound Evaluation
Solves retrylp_retries and retry_attempts in training/solver/iterations.parquet; retry_level in training/solver/retry_histogram.parquet; the LP prescaling diagnostics in training/scaling_report.jsonThe per-phase solver profiles training.solver.backward, training.solver.forward and simulation.solver; modeling.cost_scale_factor
Balances recomputed from outputs do not closelp_row_violations in training/solver/iterations.parquet and simulation/solver/iterations.parquet; the end-of-run warning of Solves that miss a row boundNone: report the case with the counts
Slow startsetup.load_seconds, setup.stochastic_fit_seconds, setup.production_fit_seconds, setup.evaporation_fit_seconds and setup.broadcast_seconds in training/metadata.json (collected on rank 0)For the stochastic fit, export the fitted artifacts once and reuse them: exports.stochastic and Exporting Stochastic Artifacts

When HiGHS returns a non-terminal error, the solver automatically escalates through a bounded retry sequence rather than surfacing the failure. The caller never sees intermediate failures — only the final solution or a single classified solver error. Escalation is capped by a built-in per-attempt iteration limit and per-level and overall wall-clock budgets, so a pathological solve cannot stall the run indefinitely.

How far escalation went on a given solve is recorded in the retry_level output column (values 0–11) and written to training/solver/retry_histogram.parquet for post-run analysis; a solve that never retries adds no histogram row; level 0 is the first retry level. The escalation strategy itself — which options each level applies — is a solver-internal detail documented in the novomodelo-solver README.


Before each stage’s LP template is built, Novomodelo applies its own geometric-mean row/column prescaler that normalizes the constraint-matrix coefficients toward 1.0, improving numerical conditioning. The prescaling mechanism and its exact-unscaling guarantee are documented in the math layer under LP Layout and Scaling.

The objective is divided by a cost-scale factor (default 1_000_000.0, configurable as modeling.cost_scale_factor) to reduce the magnitude of objective coefficients, improving simplex numerical stability.

Because the prescaler already conditions the matrix, HiGHS’s internal scaling (simplex_scale_strategy) is disabled by default in every solver profile. It is overridable per phase via the solver profile’s scale field: selecting solver_scaling re-enables solver-side scaling and forgoes the exact single-unscaling guarantee. See Solver profile fields for the field and its trade-off.

The scaling diagnostics are written to training/scaling_report.json after template construction, documenting the coefficient range before and after scaling for each stage.


As training progresses, the row pool grows and LP solve times increase. Novomodelo provides a two-stage row management pipeline (Stage 1, the selection pass, then Stage 2, the budget pass) to control this growth.

The pipeline runs after each iteration’s backward pass and cut synchronization:

Stage 1 · Strategy-based selection(check_frequency gated)Stage 2 · Budget enforcement(every iteration)

Two methods are available, configured through training.cut_selection in config.json; the Configuration reference owns the keys and defaults:

StrategySelection rule
level1At each stored trial state, the cut with the highest value owns the state (an exact tie goes to the lowest pool slot); a scored cut is active exactly when it owns a state
dynamicLazy row generation: each solve starts from a resident subset of the active cuts and grows it with the cuts the current LP solution violates; deactivates nothing

level1 respects check_frequency: the selection pass runs only at iterations that are multiples of it. It selects the pools of the stages strictly between the first and the last; the first stage’s pool and the terminal pool are never selected. Each such pool stores the trial state of every cut the run adds there, or those of the last lookback_iterations iterations, and every scored cut, current-iteration cuts and inactive cuts included, competes at every stored state, so a deactivated cut is reactivated when it later owns a state. The rule is in Level 1 Selection.

dynamic operates differently: it is a per-solve lazy loop that adds cuts on demand rather than deactivating from a pool scan, and it does not read check_frequency. See The Dynamic Method for the procedure. A solve that reaches the round limit with a candidate still violated is counted in lazy_round_cap_hits of training/convergence.parquet.

A hard cap on LP size, enabled via max_active_per_stage. It applies to the pools of the stages strictly between the first and the last, each node’s pool separately, and counts only the cuts the current run generated: cuts loaded from a policy, boundary cuts among them, neither count nor are evicted. When the count exceeds the cap after the selection pass, the pool evicts cuts of earlier iterations in ascending order of the iteration in which a cut last bound, then of the number of solves in which it bound, then of its pool slot. Cuts of the current iteration count but are never evicted. Under level1, a later selection pass reactivates an evicted cut that owns a state, and the cap is applied again in that iteration. The cap is refused with the dynamic method and with an enumerated training.selection.

Unlike the selection pass, budget enforcement runs every iteration (not gated by check_frequency).

Why it matters: High-parallelism configurations (many forward passes, few iterations) accumulate more active rows than low-parallelism configurations (fewer forward passes, more iterations), making each backward LP solve proportionally more expensive. Bounding LP size makes high-parallelism configurations viable without unbounded solve-time growth.

The row management pipeline writes per-pool statistics to training/cut_selection/iterations.parquet, where each column is defined. The knob table names the columns that diagnose backward-time growth.


A run has two levels of parallelism: MPI ranks, and worker threads inside each rank. --threads sets the number of worker threads per rank (default 1, at least 1); the launcher sets the rank count (see HPC & Cluster Deployment).

The Execution block of the run banner reports both levels. The excerpt below shows an example two-rank run with two threads per rank on one host; it omits the Solver line and writes the hostname as <host>:

Execution
Backend: MPI (MPICH 4.3.2, MPI 4.1)
Threads: Funneled, 2 rayon threads per rank
Layout: 2 ranks on <host>

The Threads: line gives the MPI thread level and the worker threads per rank; the Layout: line gives the rank count and the host. A run on several hosts shows across N nodes there, with the ranks of each host listed below it.

Under a sampled forward pass, the forward_passes trajectories of an iteration (see training.selection) are split over the ranks in contiguous blocks. Each rank takes forward_passes / ranks of them, rounded down, and the first forward_passes mod ranks ranks take one extra: 50 passes on 4 ranks give 13, 13, 12 and 12, and ranks beyond the number of passes receive none. A rank then splits its block statically over its worker threads the same way, so the assignment of trajectories to threads depends on the counts only, never on thread scheduling. Simulation scenarios are split over ranks and threads the same way.

Each rank runs the backward pass for the trial points of its own forward passes; the work units handed to its worker threads are defined under Backward-Pass Scheduling. Under an enumerated forward pass, the backward pass instead splits the successor outcomes of each node across the ranks, which allgather them so that every rank derives the same cut.

Every rank must run with the same --threads value: the backward pass of a sampled forward pass compares the worker counts across the ranks and stops the run when they differ.

By default (by_scenario), each parallel backward-pass work unit is one whole trial point. The optional by_node scheduler uses finer (trial point, opening block) work units for better load balance when trial points solve in variable time. Either way the produced cut set and lower bound are independent of worker and thread count. See parallelism.backward_scheduler for the configuration, and the novomodelo-sddp README for the scheduler and inter-rank communication internals.

The by_node scheduler shrinks the work unit from one trial point to one (trial point, opening block) pair, giving the work-stealing counter more, finer-grained units to hand out. This improves load balance specifically when trial points solve in variable time: a worker that finishes a cheap trial point quickly can pick up a block from a still-running expensive one instead of idling until an entire trial point frees up.

Within a stage, opening-blocks are claimed hardest-first: blocks are ranked by the previous iteration’s mean simplex pivot cost per (stage, block), descending, so the most expensive blocks are claimed early rather than left for whichever worker happens to reach them last. This claim order changes only which worker processes which block, and when — it is result-neutral: the produced cut set and the training lower bound are identical to canonical (whole-trial-point) claiming.

See parallelism.backward_scheduler for the config field that enables by_node and sets its block_size.

Each algorithmic phase — forward sweep, backward sweep, and simulation — can override the built-in solver profile (feasibility tolerances, the simplex update limit, pricing strategy, the scale field, and more) via training.solver.backward, training.solver.forward, and simulation.solver. See Solver profile fields for the full field list, the built-in profile each phase starts from, and the single-unscaling trade-off carried by scale.

Overriding the backward phase with tighter feasibility tolerances can reduce backward-pass solve-time variance, which improves load balance across worker threads and shortens wall-clock training time.

Under a sampled forward pass and in simulation, scenarios are assigned to workers by the static partition described under Work Partition. Within each scenario the stage LP is loaded once. Per stage, the forward-pass and simulation solves patch the incoming-state column bounds, the stochastic NCS generation column bounds, and the load-balance and inflow row bounds.

The lower-bound evaluation (solving a stage-0 LP for every opening in the tree) runs as a single-threaded serial loop on rank 0. Each opening patches correctness-critical per-opening state (e.g. NCS column bounds) on a shared solver in sequence, so the step is not parallelized.

The ranks exchange data through blocking MPI collectives, so communication does not overlap computation. MPI is initialized at the Funneled thread level, which the Threads: line of the Execution block names: only one thread of a rank makes MPI calls. An allgatherv gives every rank the data of all ranks, an allreduce combines a value over all ranks, and a broadcast sends a value from rank 0 to the others. On every run the main collectives are:

  • an allgatherv of the forward-pass costs, reduced in canonical order, so the result does not depend on the rank count;
  • allreduce sums of integer counters, among them the solve counts of the printed summary;
  • a broadcast of the lower bound, which rank 0 evaluates (see Lower Bound Evaluation) and the other ranks wait for.

Under a sampled forward pass the backward pass adds:

  • an allgatherv of the trial states of all ranks at the start of each backward stage (timed as state_exchange_ms);
  • a per-stage allgatherv of the cuts generated in the backward pass, at the end of each backward stage, which every rank waits on (timed as cut_sync_ms);
  • an allreduce sum of the per-iteration cut-binding increments.

The enumerated backward pass has neither the state exchange nor the cut exchange; it uses an allgatherv of the successor outcomes of each node instead (see Work Partition). The communicator backends are documented in the novomodelo-comm README.

Each LP solve runs on one thread: the bundled HiGHS is built without its default threads, so during the forward pass, the sampled backward pass and simulation the number of LPs a rank solves at the same time is its --threads. The enumerated backward pass runs on one worker thread per rank.

Rank 0 writes the training outputs. The timing tables and the printed Time split show rank 0’s view: the phase wall, forward wait, and serial values are rank 0’s own, not an aggregate over the ranks. The backward wait (bwd_load_imbalance_ms) is taken over the workers of every rank. The printed solve counts are sums over all ranks, and the printed solve time is the mean per worker over all ranks.

Each rank holds its own full copy of the case data, the opening tree, the cut pool, and the stage templates; ranks on one node do not share memory. The cut pool is sized at startup from the iteration budget (the largest iteration_limit in the stopping rules) times the forward passes, for every stage. The memory of every rank is set by that size, and placing more ranks on a node multiplies that node’s memory use.

Rank 0 loads the case and broadcasts the system data, the configuration, and any user-supplied opening tree; the other ranks rebuild the stochastic context and the hydro models from the case directory. Every rank must read the case directory, so it must be visible at the same path on every node (a shared filesystem or a staged copy). A resume or warm start also loads the policy checkpoint on every rank, from the output directory.


  • Configuration — row-selection and row management configuration
  • Output Format — timing, solver statistics, and row-selection output schemas
  • novomodelo-solver — solver interface and retry-escalation internals (novomodelo developer guide)
  • novomodelo-sddp — training-loop architecture, scheduling, and data structures (novomodelo developer guide)
  • novomodelo-comm — communicator backends and collectives (novomodelo developer guide)
  • ARCHITECTURE.md — novomodelo crate topology and dependency graph (novomodelo developer guide)