|
FhSim
3.1.0
Marine systems simulation
|
|
|
FhSim
3.1.0
Marine systems simulation
|
The FhSim test library provides support for simulation testing in external SimObject/plugin projects.
fhsim::test::RunTest(const TestSpec&).TestSpec is default-constructible).TestJacobianMode::{No, Yes, Auto}, default Auto).TestResult) that includes simulation output, timing, and Jacobian outcome.ReadOnly, CreateIfMissing, Create) for regression workflows.TestSpec::diagnosticsJsonPath.| Goal | API |
|---|---|
| Run a simulation test | fhsim::test::RunTest(spec) |
| Compare run vs reference files | result.CompareRegression(spec) |
| Create baseline references | result.CreateBaselineReference(spec) |
| Compare two runs directly | result.CompareRegression(otherResult) |
| Assert on output values without disk I/O | fhsim::test::CapturingObserver |
Parse the <Integrator> configuration only | fhsim::test::ReadIntegratorConfig(dirname, cfg) (returns the engine-internal IntegratorConfig) |
Set this once in your test main():
testDir (root test folder)inPath (input XML + reference files)resPath (result files)logPath (log files)runPath (working directory for simulation launch)logLevelToScreen, logLevelToLogreferenceMode (ReadOnly by default)For project test binaries, expose this config through a helper such as GetTestRuntimeConfig() and use it in tests.
For ID ImplicitIntegration_Euler1imp, TestSpec resolves paths as:
ImplicitIntegration (part before first _)<inPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_in.xml<resPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_res.txt<logPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_log.txt<inPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp.ref<inPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_ref.csvIf you pass pathPrefix to TestSpec, paths resolve under:
<pathPrefix>/in/...<pathPrefix>/out/...TestSpec::jacobianMode controls Jacobian verification inside RunTest:
TestJacobianMode::No → Jacobian check disabledTestJacobianMode::Yes → Jacobian check required (hard-fails if unavailable or failing)TestJacobianMode::Auto (default) → run when Jacobians are available, otherwise skipOptional Jacobian checker tuning:
Use ReferenceMode::CreateIfMissing during bootstrap and ReadOnly in CI.
CompareRegression(spec) uses two artifacts per test ID:
<ID>.ref<ID>_ref.csvFor test ID MyTest (group subdir MyTest):
<runtimeConfig.inPath>/MyTest/MyTest.ref<runtimeConfig.inPath>/MyTest/MyTest_ref.csvPerformance file (.ref) is a line-based key-value format:
#<key> <value>Supported keys:
cpu_ref (double): reference CPU time in seconds (informational only)wall_max (double): maximum allowed wall-clock time in seconds (hard limit)output_rms_max (double): maximum RMS error per output column vs _ref.csv_ref.csv uses the same semicolon-separated output format as standard simulation output.
The old constructor-based style still works:
The recommended style is explicit field assignment on a default-constructed struct:
jacobianMode, referenceModeOverride) without long constructor chains.TestSpec(id, cfg, simTime, maxRunTime, pathPrefix)spec.ID = idspec.runtimeConfig = cfgspec.simTime = simTimespec.maxRunTime = maxRunTimespec.pathPrefix = pathPrefixReadOnly: compare only, never write references.CreateIfMissing: write only missing reference artifacts, then compare.Create: always regenerate reference artifacts from current run.Resolution order:
TestSpec::referenceModeOverride (if set)TestRuntimeConfig::referenceModeCapturingObserver collects simulation snapshots in memory and provides named lookup for both output port values and ODE state values. Use it when a test needs to assert directly on simulation data without a result file on disk.
RunTest and the CSV-based regression workflow remain the right choice for regression tests. CapturingObserver is for tests that assert on specific numeric values step-by-step, or that want to skip file I/O entirely.
CapturingObserver is an IStateObserver. Register it with a SimulationManager built from the same input file RunTest would use:
If the test also needs regression comparison, keep ctx.outputPath = spec.outFile.string(). The FileOutputObserver and CapturingObserver coexist without conflict.
CapturingObserver exposes two lookup methods:
Use OutputAt for most tests. The XML output spec is the declared observable interface of the model. Output names (objectName, signalName) are stable across XML reorganisation. OutputAt accesses the same values that result.output and CompareRegression use, making it consistent with the regression workflow.
Use StateAt when:
<Output> element yet.StateAt is keyed by the objectName and tagName strings that the SimObject passes to ISimObjectCreator::AddState(). These are stable as long as the SimObject's state registration does not change, but they can be affected by reordering SimObjects in the XML.
| Situation | Use |
|---|---|
| XML declares an output column for this value | OutputAt / FinalOutput |
| Asserting on a value not in the XML output spec | StateAt / FinalState |
| Regression comparison across runs | RunTest + CompareRegression |
| Checking that the simulation completes without error | result.ok from RunTest |
The step index (0-based) runs over output steps only — internal integrator steps that do not produce output are not counted, matching the cadence of the CSV result file.
TestSpec: ID, timing limits, runtime config snapshot, resolved paths, Jacobian policy, optional diagnosticsJsonPath.TestResult: ok/error, timing metrics, resolved paths, parsed output, Jacobian run result.JacobianRunResult (in TestResult::jacobian): status (NotRequested, SkippedUnavailable, Passed, Failed, Error), diagnostics, and per-scope summaries.PerformanceMetrics: wall/cpu/sim-time/step counters and rates.RegressionReport: ok + accumulated diagnostic text.ResultFileData / ObjectTimeSeries: parsed output columns.FindSeries(result, objectName, tagName): locate a specific output column quickly.CapturingObserver: in-memory observer with OutputAt/StateAt/FinalOutput/FinalState accessors.Example:
Set TestSpec::diagnosticsJsonPath to persist unified diagnostics data after a successful RunTest(spec) call.
RunTest writes this file only when diagnostics are enabled in the simulation input (for example by adding <Diagnostics/> under <Integrator>).
The file contains:
metadata: state and port naming metadata (simObjectNames, stateNames, portObjectNames, portNames, portIndices).snapshots: by default the latest snapshot; full history when history accumulation is enabled by the integrator/diagnostics pipeline.For tests that need to verify a SimObject's analytical Jacobian independently of the RunTest pipeline, use CheckJacobians or CheckJacobiansFromXml from JacobianChecker.h (included transitively by FhsimTest.h and TestSpec.h):
Each element of the returned vector covers one *(state-point × scope)* combination, where scope is "system" plus one entry per SimObject that returns HasJacobians() == true.
The state points are:
randomSamples perturbations of it, at every time in timePoints. A time point changes only the time argument, not the state, so it does not follow the simulation;k/trajectorySamples of its time span, k = 1..trajectorySamples, each checked at the time actually reached. The span ends at trajectoryStopTime when it is set; RunTest sets it to TestSpec::simTime. A Jacobian term that only switches on once the dynamics have moved — a contact, a cable going taut, a surface crossing, a saturating controller — is compared here and nowhere else. A trajectory that cannot be integrated gives an error result;shiftStateOffsets, all of the above again for the per-SimObject checks only, in a copy of the scenario with the neutral SimObject Testtools/StateOffsetPadding placed first, so that every object sits at a non-zero state offset. OdeJacobian is handed its own block and its own slice of the state, both indexed from zero; a callback that indexes them with the global offset AddState returned is right only for the model's first stateful object and fails here. The padding object comes from the fhsim_testtools_objects library, which the FhSim package installs into bin/SimObjectLibraries; a model library's playpen imports it with the rest of FhSim's bin. A write past the object's own block is reported as an error result instead of corrupting memory.scopeOverrides maps the name of a SimObject with states, or "system", to a JacobianScopeOverride {skip, relTol, absTol} for a SimObject whose Jacobian is partial by design. The override applies to that object's own block and to every entry of the system matrix whose row or column is one of its states, so the rest of the system matrix, and every other object, is still checked at the global tolerances. An override only loosens: a tolerance tighter than the one an entry would otherwise get has no effect, and an entry whose row and column carry different tolerances is compared at the larger. A skipped scope passes with skipped set, and every result counts the entries it left out (exemptedEntries) or compared at a tolerance an override loosened beyond the global one (relaxedEntries); RunTest lists each exempted scope in JacobianRunResult::diagnostics and in its JacobianScopeSummary. A key that names no SimObject of the scenario, or a SimObject without states (which owns no rows or columns and so would exempt nothing), gives an error result.
JacobianCheckResult field | Type | Meaning |
|---|---|---|
ok | bool | All elements within tolerance |
label | string | Human-readable scope label |
scope | string | "system" or SimObject name |
sampleIndex | int | 0 = IC, >0 = random perturbation index, -1 = trajectory sample |
timeIndex | int | Index into config.timePoints, -1 for a trajectory sample |
trajectorySample | int | 0 = not a trajectory sample, k > 0 = the k-th state reached by integrating |
stateOffsetShift | int | States the padding object put in front; 0 = the scenario as written |
skipped | bool | A scope override skipped the whole scope (ok is then true) |
exemptedEntries | int | Elements a scope override left uncompared |
relaxedEntries | int | Elements compared at a tolerance a scope override loosened beyond the global one |
atTime | double | Simulation time of the check |
nStates | int | Jacobian block dimension |
maxAbsErr | double | Worst absolute error across all elements |
maxRelErr | double | Worst relative error across all elements |
mismatchCount | int | Number of elements exceeding tolerances |
violations | vector<JacobianElementError> | Up to maxViolationsReported worst elements |
error | string | Non-empty on setup/runtime failure |
Summary() | — | Formatted multi-line diagnostic string |
Each JacobianElementError carries row, col, analytical, numerical, absErr, and relErr.
ISimObjectCreator::AddState returns an index into the whole model's state vector, and the X and XDot buffers OdeFcn is handed span the whole model. A SimObject that indexes them from zero reads and writes the states of whichever object owns the model's first indices. Nothing in a simulation notices: the derivative it overwrote simply becomes wrong, and in a model with a single stateful SimObject the mistake has no effect at all — which is why it survives until a second object is added.
CheckStateBlocks runs each SimObject's OdeFcn alone over a derivative buffer prefilled with a sentinel and reports every element written outside that object's own block:
Check a model with at least two stateful SimObjects: an object that owns the model's first states cannot write outside its own block by indexing from zero, so a single-object model proves nothing.
StateBlockCheckResult field | Type | Meaning |
|---|---|---|
ok | bool | The SimObject wrote nothing outside its own states |
simObjectName | string | Name of the checked SimObject |
stateOffset | int | Its first global state index |
stateCount | int | Number of states it owns; 0 when stateless |
violationCount | int | Out-of-block writes seen, before the report cap |
violations | vector<StateBlockViolation> | Up to maxViolationsReported of them |
Summary() | — | Formatted multi-line diagnostic string |
A model that cannot be built, or that cannot evaluate one SimObject at a time, throws std::runtime_error; there is no error field to test, so a result that comes back always describes a SimObject that was checked.
Each StateBlockViolation names the stateIndex written, the ownerName of the SimObject whose derivative it overwrote, the value written, and the sampleIndex and atTime where it was seen.
The initial-condition half of the same contract needs no test: the assembler checks it itself, around every SimObject::InitialConditionSetup call, and reports a stray write as an error.
The mirror image of the contract above. OutputPortJacobian, InputPortJacobian and InputOutputJacobian are handed a buffer sized for one block and a state pointer already offset to the SimObject's own states, so every index they use is local — while the index AddState returned is global. An object that uses the latter writes outside the buffer it was given, which is correct only while it is the model's first stateful SimObject.
The assembler's workspaces are sized for the largest block in the model, so such a write lands either inside the workspace, where the next link silently consumes it, or past the workspace, where it corrupts the heap. Neither is visible to Jacobian verification, which sees only that some number came out wrong.
CheckPortJacobianBlocks calls every callback the assembler would call, over a buffer padded with a sentinel-filled guard region, and reports every guard cell that moved:
Check a model in which the SimObject under test is not the first stateful object, and in which its ports are really connected: only a connected, fully analytical link makes the assembler call these callbacks, so a model whose chain runs through a SimObject without port Jacobians reports nothing at all.
PortJacobianCheckResult field | Type | Meaning |
|---|---|---|
ok | bool | The callback wrote only inside the buffer it was given |
simObjectName | string | SimObject whose callback was called |
portName | string | Port it was called for; "Out<-In" for InputOutputJacobian |
callbackName | string | OutputPortJacobian, InputPortJacobian or InputOutputJacobian |
rowCount, columnCount | int | Dimensions of that buffer |
violationCount | int | Out-of-buffer writes seen, before the report cap |
violations | vector<PortJacobianViolation> | Up to maxViolationsReported of them |
Summary() | — | Formatted multi-line diagnostic string |
One result covers one (SimObject, port, callback) triple across every link and every state point that made the assembler call it.
Two limits are worth knowing. A write into the wrong cell inside the buffer is not an out-of-buffer write and is not reported here — that is what the Jacobian check above is for, and it catches it as a mismatch against the finite difference. And the guard region is sized for the mistake this exists to catch, an index built from a global state index; a wilder write than that runs off the end of the guard undetected.
To suppress log output in tests, use the platform-portable null device helper:
This avoids hard-coding platform-specific paths in test code.
| Symptom | Likely cause | Action |
|---|---|---|
| Missing input file | ID-to-path mapping mismatch | Verify test ID and <inPath>/<group>/<ID>_in.xml |
Missing .ref in ReadOnly mode | Baseline not created | Run once with CreateIfMissing or Create |
Missing _ref.csv with output_rms_max | Output reference absent | Generate baseline output reference |
| Wall-time budget exceeded | maxRunTime too low or simulation regressed | Raise budget for heavy tests or investigate performance |
| Output identity mismatch | Changed output column names/order | Update reference and/or adapt test expectations |
Jacobian check skipped in Auto | Model does not expose Jacobians | This is expected in Auto; use Yes to require Jacobians |
Jacobian hard-fail in Yes mode | Jacobians unavailable or numerical/analytical mismatch | Inspect result.jacobian.diagnostics and failing scope summaries |
1.9.8