Policy Checkpoint
The policy checkpoint is the directory under policy.path (policy/ by default)
that novomodelo run writes after training. It holds the pool-keyed cuts/NNN.bin
files, the node-keyed basis/NNN.bin files, the pool-keyed states/NNN.bin
trial states (for each pool that stores them, under exports.states or the
level1 selection method), and manifest.bin, written last. A run that fails
writes its checkpoint under <policy.path>.partial instead (see
A failed run).
policy/cuts/NNN.bin
Seção intitulada “policy/cuts/NNN.bin”FlatBuffers binary file encoding all cuts for a single pool. Pool-keyed:
one file per pool (cuts/<pool_id>.bin), zero-padded to three digits (e.g.
000.bin, 012.bin); a pool shared by several leaf nodes appears once. On a
plain stage chain, pool id equals stage index. The general node -> pool mapping
(for a branching policy graph) is resolved through the nodes[].pool_id
entries of policy/manifest.bin, never trusted from the
file name.
Methodology: Cut Management
Each cuts/<pool>.bin is self-describing: alongside its cuts it carries its
own state_dimension, its own cost_scale_factor (the authoritative load-time
scale for this pool — see Cost-Scale Canonicalization),
and its own graph identity (node and graph-stage ids), so no field of the
study-global manifest is needed to interpret a pool’s coefficients.
StageCuts header fields (the root table of each cuts/<pool>.bin):
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
stage_id | uint32 | No | — | Pool id (0-based), the key that names cuts/<pool>.bin. |
state_dimension | uint32 | No | — | The writing study’s state-vector length. A pool that projects fewer dimensions holds shorter coefficient vectors, of the length of its entity_manifest. |
capacity | uint32 | No | — | Total preallocated cut slots in the pool. |
warm_start_count | uint32 | No | — | Protected-prefix length: the leading slots the writing run started with, loaded from a previous policy; 0 for a fresh run, except in the terminal pool of a run that loads policy.boundary, where it counts the boundary cuts. |
cuts | [AffinePiece] | No | — | The pool’s cuts, one AffinePiece per cut (fields below), strictly ascending by slot_index. |
populated_count | uint32 | No | — | Number of filled slots; equals the length of cuts. |
entity_manifest | [EntitySlot] | No | — | Per-slot entity identity, one EntitySlot per state dimension (fields below). |
cost_scale_factor | float64 | No | — | The scale of every cut’s intercept and coefficients: the objective cost-scale factor of the writing study, whose scaled units they are in (see Cost-Scale Canonicalization). |
node_id | int32 | No | — | Policy-graph id of the pool’s sole owning node; -1 for a shared pool. |
graph_stage_id | int32 | No | — | Graph-stage id of the stage that owns the pool; -1 when unresolved. No load reads it: a boundary load selects its source pool by priced_state_date. |
priced_state_date | int32 | No | — | The owning stage’s end_date encoded YYYYMMDD (year*10000 + month*100 + day); -2147483648 (the int32 minimum) when not recorded. |
The binary is not human-readable. The logical record structure of each cut
(one AffinePiece per entry in cuts) is:
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
slot_index | uint32 | No | — | The cut’s slot in the pool, which a stored basis names in piece_row_slots. Required for checkpoint reproducibility and basis warm-starting. |
iteration | uint32 | No | — | Training iteration that generated this cut. |
forward_pass_index | uint32 | No | — | Forward pass index within the generating iteration. |
intercept | float64 | No | — | Pre-computed cut intercept , where is the state at the generating forward pass node and the aggregated stage value there; in the scale of the pool’s cost_scale_factor. |
coefficients | [float64] | No | — | The cut’s slope: one coefficient per state dimension, the sensitivity of the future cost to each incoming state value (see Cut Management). Length equals the state dimension — the number of entries in the embedded entity manifest below, also the pool’s own state_dimension (per-pool, not a policy/manifest.bin field). |
is_active | bool | No | — | Whether this cut was active in the LP when the checkpoint was written: the last selection. Inactive cuts are retained for potential reactivation by the selection method. |
scored | bool | No | — | Whether the writing run stored the trial state the cut was computed at (under exports.states or the level1 method). Kept when that state is later dropped from states/; a level1 run that loads the policy selects a scored cut like its own. |
active_count | uint64 | Yes | — | The writing run’s count of the cut’s binding activity; absent unless the manifest’s activity_counters is set. |
last_active_iteration | uint64 | Yes | — | The writing run’s last iteration in which the cut was binding; absent unless the manifest’s activity_counters is set. |
The wire tables keep the ids of fields that format version 4 removed —
AffinePiece.piece_id and StageCuts.active_cut_indices among them — as
reserved slots that no writer fills; each cut’s is_active carries the active
set.
Each NNN.bin also embeds a per-slot entity manifest — one entry per
state-vector dimension, in canonical cut-coefficient order — recording which
entity each coefficient position belongs to. It is what policy-load
validation checks (see Policy Management):
a policy whose dimensions match the current study by count but bind to different
entities is rejected.
EntitySlot fields (one entity_manifest entry per state dimension):
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
entity_type | EntityType | No | — | byte enum: state-dimension kind — 0 HydroStorage, 1 HydroInflowLag, 2 AnticipatedThermalState, 3 HydroTransitBucket. |
entity_id | int32 | No | — | Owning entity id — for a transit bucket, the downstream hydro. |
subindex | uint32 | No | — | Secondary index within the entity: the inflow-lag order, anticipated-commitment ring slot (counted from stage 0: for the Hold Ring slot ), or transit maturity lag. |
was_active | bool | No | — | Whether the owning entity was operationally active at this stage (excluded from the load-time identity check). |
reference_date | int32 | No | — | The HydroInflowLag reference past-stage start_date, encoded YYYYMMDD (year*10000 + month*100 + day); sentinel -2147483648 (the int32 minimum) for every other family. |
interval_start | int32 | No | — | Inclusive start of the forward-family half-open window, encoded YYYYMMDD: a HydroTransitBucket arrival start or an AnticipatedThermalState delivery start. Sentinel -2147483648 for HydroStorage and HydroInflowLag, and for an AnticipatedThermalState ring slot that holds no live commitment at the pool’s stage or whose delivery lands past the study calendar extended by the post-study stages. |
interval_end | int32 | No | — | Exclusive end of the window, paired with interval_start; reads the sentinel -2147483648 wherever interval_start does. |
Wire readability. A reader parses policy/manifest.bin first, requires the
CBVF file identifier and a format_version of 4, and only then decodes the
payloads. There is no conversion between format versions; see
Versioning policy.
Load admission. Every policy load requires the software and
software_version recorded in manifest.bin to equal the running software’s
name and version exactly (see the version gate).
policy/basis/NNN.bin
Seção intitulada “policy/basis/NNN.bin”FlatBuffers binary file encoding the LP simplex bases stored at a single
policy-graph node, one per scenario that held one. Node-keyed: files are
named by the node’s 0-based position (basis/NNN.bin; the position is the
stage_id field below), zero-padded to three digits, and exist only for nodes
training captured a basis for; on a stage chain the position is the 0-based
stage position, one file per stage. Bases are always written with the
end-of-training checkpoint, and the basis/ directory is written even when
no node holds a basis; a load reads a checkpoint without basis/ as one with
no stored bases, so a checkpoint unpacked from an archive that dropped the
empty directory still loads. Warm-start, resume, and simulation-only loads read
the bases to warm-start the stage LP solves, using each basis that fits its stage LP
and leaving the others out (see the
stored-basis gate); boundary
loads read none.
Methodology: LP Warm-Start
The logical record structure (the StageBasis root table) is:
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
stage_id | uint32 | No | — | 0-based position of the policy-graph node (equal to the 0-based stage position on a stage chain), not the declared stage id. |
iteration | uint32 | No | — | Training iteration that produced these bases. |
entries | [BasisEntry] | No | — | One entry per global scenario holding a basis at this node, ascending by scenario (fields below). |
BasisEntry fields (one entries element per scenario):
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
scenario | uint32 | No | — | Global scenario index holding this basis. |
column_status | [uint8] | No | — | One canonical basis-status code per LP column (variable). See the encoding note below. |
row_status | [uint8] | No | — | One canonical basis-status code per LP row (constraint). See the encoding note below. |
piece_row_slots | [uint32] | No | — | Pool slot of each trailing cut row of row_status: entry i is the slot_index of the cut in the i-th cut row. |
The column_status/row_status bytes are the canonical solver-neutral
BasisStatus superset 0..=6 — 0 Lower, 1 Basic, 2 Upper, 3 Zero,
4 Nonbasic, 5 Superbasic, 6 Fixed — not a backend-specific encoding; novomodelo
maps each LP backend (HiGHS default, CLP opt-in) onto this canonical set on
export. The lengths of the status vectors are the basis’s column and row
counts; the rows before the last len(piece_row_slots) are the template rows.
A training load (resume or warm start) gives each scenario its own entry; a
scenario without one starts cold on resume and takes the node’s lowest-scenario
entry that fits on a warm start. A simulation-only load warm-starts every
scenario from the node’s lowest-scenario entry that fits. The manifest’s basis_store_complete records
whether basis/ holds the bases of every scenario the run gathered.
policy/states/NNN.bin
Seção intitulada “policy/states/NNN.bin”FlatBuffers binary file encoding the trial states one pool stores, one per
stored cut slot. Pool-keyed: one file per pool that stores trial states
(states/<pool_id>.bin, the same key as cuts/<pool_id>.bin), zero-padded to
three digits. With exports.states set, every pool that receives cuts stores
the states of its own cuts: the first stage’s pool keeps every state, and each
pool strictly between the first and the last stage keeps those inside
lookback_iterations when that key is set. A level1 run stores and writes the
states of the pools it selects whether or not exports.states is set. The
states/ directory is omitted when no pool stores states. Like the cut files,
each NNN.bin embeds the per-slot entity manifest describing its state
dimensions.
Methodology: Cut Management
A trial state is the state vector a cut was computed at. A resume reads these
files to restore the trial states a level1 run selects at; a warm_start
loads the state stored for a scored cut with that cut. The manifest’s
state_history records which states the interior pools hold, and
total_visited_states their total count.
StageStates fields (the root table of each states/NNN.bin):
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
stage_id | uint32 | No | — | Pool id (0-based), the same key as the stage_id of cuts/<pool>.bin. |
node_id | int32 | No | — | Declared id of the pool’s sole owning node; sentinel -1 for a shared pool or a buffer that omits the field. |
state_dimension | uint32 | No | — | Length of each state vector: the length of the pool’s entity_manifest in cuts/<pool>.bin. |
count | uint32 | No | — | Number of state vectors stored for this pool. |
data | [float64] | No | — | Flat array of count * state_dimension elements, row-major (one state per row). |
entity_manifest | [EntitySlot] | No | — | Per-slot entity identity of each state dimension; state_dimension entries, with the fields of the EntitySlot table under policy/cuts/NNN.bin. |
slots | [uint32] | No | — | Pool slot of each stored row, strictly ascending; count entries. Each names a cut of the pool whose scored is set. |
policy/manifest.bin
Seção intitulada “policy/manifest.bin”The study-global checkpoint manifest: a FlatBuffers root (the
CheckpointManifest table, file_identifier "CBVF") describing the checkpoint
at a high level, written last as the commit signal and read first behind
the format_version gate. Every study-global fact a load needs — the study graph,
the stage count, and the producer provenance — lives here. On the wire,
CheckpointManifest is one flat table: the graph and producer fields sit at its
root, beside a nested season_manifest. The Python bindings group them under
graph_manifest, producer and season_manifest (see
load_policy); the tables below
follow that grouping, with the same field names.
Methodology: Policy Graphs
Python view of the manifest
Seção intitulada “Python view of the manifest”Python dict top-level fields:
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
format_version | integer | No | — | On-disk format version; must equal 4 on read, before any payload is parsed (wire readability). Admission is the separate software and version check. |
software | string | Yes | — | Name of the software that wrote this checkpoint: "novomodelo" from every novomodelo writer, None for a buffer without the field (wire id 20). The version gate compares it, and novomodelo.write_policy_checkpoint stamps it. |
software_version | string | No | — | Version of that software, compared verbatim (wire id 1); every policy load requires it to equal the running version. |
created_at | string | No | — | ISO 8601 timestamp when the checkpoint was written. |
num_stages | integer | No | — | Number of stages the graph manifest spans. Must match the case configuration on resume. |
graph_manifest | object | No | — | Graph manifest: node list, edge list and pool-set size; each node names its pool (nodes[].pool_id). |
producer | object | No | — | Producer-namespaced metadata — the training algorithm’s own recorded state. See below. |
season_manifest | object | No | — | Study-global season / periodic-AR compatibility descriptor: cycle_code (0 monthly, 1 weekly, 2 custom, 255 absent: no season map declared), n_seasons, and per-hydro AR orders (ascending by hydro id). Compared on boundary loads only — see Compatibility requirements; full-FCF loads do not compare it. |
graph_manifest fields:
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
graph_manifest.n_pools | integer | No | — | Number of distinct pools (the pool-set size). |
graph_manifest.nodes[] | array | No | — | Every node, in canonical order, each with its stage and pool. See nodes[] fields below. |
graph_manifest.edges[] | array | No | — | Every directed edge with its transition probability. See edges[] fields below. |
nodes[] fields (one entry per policy-graph node):
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
graph_manifest.nodes[].id | integer | No | — | Node id: the declared node id, or, on a stage chain, the 0-based stage position (not the declared stage id). |
graph_manifest.nodes[].stage_id | integer | No | — | Declared stage id of the stage this node sits at. |
graph_manifest.nodes[].pool_id | integer | No | — | Pool whose payload holds this node’s affine pieces (the node -> pool map: leaf nodes sharing a pool all name the same pool_id). |
edges[] fields (one entry per directed policy-graph edge):
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
graph_manifest.edges[].source_id | integer | No | — | Source node id. |
graph_manifest.edges[].target_id | integer | No | — | Target node id. |
graph_manifest.edges[].probability | number | No | — | Transition probability P(source -> target). |
producer fields:
| Name | Type | Nullable | Units | Description |
|---|---|---|---|---|
producer.completed_iterations | integer | No | — | Number of training iterations completed at checkpoint time. |
producer.final_lower_bound | number | No | — | Lower bound value after the final completed iteration. |
producer.lower_bound_history[] | array | No | — | The lower bound after each recorded iteration, oldest first, the last at completed_iterations; empty when the writer recorded none. Resume restores it. |
producer.best_upper_bound | number | Yes | — | The final completed iteration’s upper bound, if available — the last value, not a min-tracked/observed best. novomodelo run always records it; null only in a checkpoint whose writer recorded none. |
producer.max_iterations | integer | No | — | Maximum iterations configured for the run. |
producer.forward_passes | integer | No | — | Number of forward passes per iteration configured for the run. |
producer.warm_start_cuts | integer | No | — | Largest per-pool warm-start cut count: the maximum of warm_start_counts[], not their sum. 0 for a fresh run without policy.boundary. |
producer.warm_start_counts[] | array | No | — | Per-pool warm-start cut counts, in pool-id order. Supersedes warm_start_cuts for per-pool accuracy when non-empty. |
producer.rng_seed | integer | No | — | RNG seed used by the scenario sampler. Required for reproducibility. |
producer.total_visited_states | integer | No | — | Total number of trial states the pools store, summed over the pools: the count of every states/<pool>.bin. 0 when no pool stores states. |
producer.training_block_mode | string | No | — | Block mode the artifact was trained under: the shared lowercase mode ("parallel"/"chronological") when every stage agrees, else "mixed". |
producer.training_block_mode_per_stage[] | array | No | — | Per-study-stage training block modes, in study-stage order. Empty when uniform; populated only for mixed-mode studies. |
producer.cost_scale_factor | number | Yes | — | Objective cost-scale factor the writing study resolved (modeling.cost_scale_factor): the scale of the writer’s scaled units, which every cut’s intercept and coefficients are in, recorded here as study-global provenance. Each pool carries its own copy in cuts/<pool>.bin, which is what a load reads; every load path requires it, and a resolved pool missing it is rejected (“predates self-describing cuts …”). See Cost-Scale Canonicalization for the details. |
producer.state_history | string | integer | Yes | — | Which trial states the pools strictly between the first and the last stage store: None (none), "whole" (every one), or a positive integer w (those of the last w iterations). On the wire, state_history_scope (0 none, 1 window, 2 whole) and state_history_window. A resume checks it against the run’s level1 setting. |
producer.activity_counters | bool | No | — | Whether every cut of every pool carries active_count and last_active_iteration; set by a run with max_active_per_stage or the dynamic method, whose resume reads them. |
producer.basis_store_complete | bool | No | — | Whether basis/ holds the bases of every scenario the writing run gathered from all ranks. Unset when the run failed, which keeps the writing rank’s own; a resume refuses a checkpoint where it is unset. |