Test Tools Library

Table of Contents

• What this gives you
• Linking
• Core API (purpose-first)
• Runtime configuration (TestRuntimeConfig)
• Path conventions and test IDs
• Recommended workflow
  • 1) Define test with struct defaults (new style)
  • 2) Jacobian policy per test
  • 3) Regression check against reference artifacts
  • 4) Baseline creation flow
  • Reference file schema
• Migration guide: constructor style to struct style
  • Why migrate
  • Quick mapping
• Reference modes
• Programmatic output capture (CapturingObserver)
  • Wiring it in
  • OutputAt vs StateAt
  • Decision table
  • Convenience methods
• Main data structures
  • Optional diagnostics JSON output
• Standalone Jacobian checking
• Utility: NullLogFile
• Troubleshooting

The FhSim test library provides support for simulation testing in external SimObject/plugin projects.

What this gives you

• One main execution entry point: fhsim::test::RunTest(const TestSpec&).
• Struct-based test definition with sensible defaults (TestSpec is default-constructible).
• Built-in Jacobian policy per test (TestJacobianMode::{No, Yes, Auto}, default Auto).
• Unified result object (TestResult) that includes simulation output, timing, and Jacobian outcome.
• Built-in reference handling (ReadOnly, CreateIfMissing, Create) for regression workflows.
• Optional diagnostics JSON artifact per test via TestSpec::diagnosticsJsonPath.

---

Linking

find_package(FhSim REQUIRED)
 
add_executable(my_tests MyTests.cpp RunTests.cpp)
target_link_libraries(my_tests
    PRIVATE
        FhSim::fhsim
        FhSim::fhsim_test
        GTest::GTest)

Core API (purpose-first)

Goal	API
Run a simulation test	fhsim::test::RunTest(spec)
Compare run vs reference files	result.CompareRegression(spec)
Create baseline references	result.CreateBaselineReference(spec)
Compare two runs directly	result.CompareRegression(otherResult)
Assert on output values without disk I/O	fhsim::test::CapturingObserver
Parse the <Integrator> configuration only	fhsim::test::ReadIntegratorConfig(dirname, cfg) (returns the engine-internal IntegratorConfig)

Runtime configuration (TestRuntimeConfig)

Set this once in your test main():

• testDir (root test folder)
• inPath (input XML + reference files)
• resPath (result files)
• logPath (log files)
• runPath (working directory for simulation launch)
• logLevelToScreen, logLevelToLog
• referenceMode (ReadOnly by default)

For project test binaries, expose this config through a helper such as GetTestRuntimeConfig() and use it in tests.

Path conventions and test IDs

For ID ImplicitIntegration_Euler1imp, TestSpec resolves paths as:

• subdir = ImplicitIntegration (part before first _)
• input = <inPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_in.xml
• result = <resPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_res.txt
• log = <logPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_log.txt
• perf ref = <inPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp.ref
• output ref = <inPath>/ImplicitIntegration/ImplicitIntegration_Euler1imp_ref.csv

If you pass pathPrefix to TestSpec, paths resolve under:

• <pathPrefix>/in/...
• <pathPrefix>/out/...

Recommended workflow

1) Define test with struct defaults (new style)

#include "FhsimTest.h"
 
TEST(MyModel, Runs)
{
    fhsim::test::TestSpec spec;
    spec.ID            = "MyModel_caseA";
    spec.simTime       = 5.0;
    spec.maxRunTime    = 30.0;
    spec.runtimeConfig = GetTestRuntimeConfig();
    // spec.jacobianMode defaults to Auto
 
    const auto result = fhsim::test::RunTest(spec);
 
    ASSERT_TRUE(result.ok) << result.error;
    EXPECT_GT(result.stepCount, 0u);
    EXPECT_FALSE(result.output.m_objectStates.empty());
}

2) Jacobian policy per test

TestSpec::jacobianMode controls Jacobian verification inside RunTest:

• TestJacobianMode::No → Jacobian check disabled
• TestJacobianMode::Yes → Jacobian check required (hard-fails if unavailable or failing)
• TestJacobianMode::Auto (default) → run when Jacobians are available, otherwise skip

Optional Jacobian checker tuning:

fhsim::test::TestSpec spec;
spec.ID            = "ImplicitIntegration_DIRK_VanDerPol_mu0";
spec.simTime       = 10.0;
spec.runtimeConfig = GetTestRuntimeConfig();
spec.jacobianMode  = fhsim::test::TestJacobianMode::Yes;
 
fhsim::test::JacobianCheckConfig jacCfg;
jacCfg.randomSamples = 2;
jacCfg.relTol = 1e-5;
spec.jacobianConfigOverride = jacCfg;

3) Regression check against reference artifacts

TEST(MyModel, NoRegression)
{
    fhsim::test::TestSpec spec;
    spec.ID            = "MyModel_caseA";
    spec.simTime       = 5.0;
    spec.maxRunTime    = 30.0;
    spec.runtimeConfig = GetTestRuntimeConfig();
 
    const auto result = fhsim::test::RunTest(spec);
    const auto report = result.CompareRegression(spec);
 
    EXPECT_TRUE(report.ok) << report.error;
}

4) Baseline creation flow

Use ReferenceMode::CreateIfMissing during bootstrap and ReadOnly in CI.

fhsim::test::TestSpec spec;
spec.ID                    = "MyModel_caseA";
spec.runtimeConfig         = GetTestRuntimeConfig();
spec.referenceModeOverride = fhsim::test::ReferenceMode::CreateIfMissing;

Reference file schema

CompareRegression(spec) uses two artifacts per test ID:

1. Performance reference file: <ID>.ref
2. Output reference file: <ID>_ref.csv

For test ID MyTest (group subdir MyTest):

• Performance file: <runtimeConfig.inPath>/MyTest/MyTest.ref
• Output reference file: <runtimeConfig.inPath>/MyTest/MyTest_ref.csv

Performance file (.ref) is a line-based key-value format:

• comments start with #
• blank lines are ignored
• key-value lines are <key> <value>

Supported keys:

• cpu_ref (double): reference CPU time in seconds (informational only)
• wall_max (double): maximum allowed wall-clock time in seconds (hard limit)
• output_rms_max (double): maximum RMS error per output column vs _ref.csv

_ref.csv uses the same semicolon-separated output format as standard simulation output.

Migration guide: constructor style to struct style

The old constructor-based style still works:

const auto& cfg = GetTestRuntimeConfig();
auto spec = fhsim::test::TestSpec("MyModel_caseA", cfg, 5.0, 30.0);
auto result = fhsim::test::RunTest(spec);

The recommended style is explicit field assignment on a default-constructed struct:

fhsim::test::TestSpec spec;
spec.ID            = "MyModel_caseA";
spec.simTime       = 5.0;
spec.maxRunTime    = 30.0;
spec.runtimeConfig = GetTestRuntimeConfig();
 
auto result = fhsim::test::RunTest(spec);

Why migrate

• Easier to read/edit in tests (named fields).
• Easier to add optional policies per test (jacobianMode, referenceModeOverride) without long constructor chains.
• Better defaults with less boilerplate.

Quick mapping

• TestSpec(id, cfg, simTime, maxRunTime, pathPrefix)
  • spec.ID = id
  • spec.runtimeConfig = cfg
  • spec.simTime = simTime
  • spec.maxRunTime = maxRunTime
  • spec.pathPrefix = pathPrefix

Reference modes

• ReadOnly: compare only, never write references.
• CreateIfMissing: write only missing reference artifacts, then compare.
• Create: always regenerate reference artifacts from current run.

Resolution order:

1. TestSpec::referenceModeOverride (if set)
2. otherwise TestRuntimeConfig::referenceMode

Programmatic output capture (CapturingObserver)

CapturingObserver collects simulation snapshots in memory and provides named lookup for both output port values and ODE state values. Use it when a test needs to assert directly on simulation data without a result file on disk.

RunTest and the CSV-based regression workflow remain the right choice for regression tests. CapturingObserver is for tests that assert on specific numeric values step-by-step, or that want to skip file I/O entirely.

Wiring it in

CapturingObserver is an IStateObserver. Register it with a SimulationManager built from the same input file RunTest would use:

#include <fhsim/testtools/FhsimTest.h>           // pulls in CapturingObserver.h
#include "SimulationManagerFactory.h"            // CreateSimulationManagerFromFile
 
TEST(SpringMass, FinalPositionIsZero)
{
    fhsim::test::TestSpec spec;
    spec.ID            = "SpringMass";
    spec.simTime       = 10.0;
    spec.runtimeConfig = GetTestRuntimeConfig();
    spec.ResolvePaths();
 
    fhsim::SimulationContext ctx;
    ctx.logPath  = spec.logFile.string();
    ctx.logLevel = spec.runtimeConfig.logLevelToLog;
    // Omit ctx.outputPath to skip FileOutputObserver (no CSV written).
 
    auto manager = fhsim::CreateSimulationManagerFromFile(
        spec.inFile.string(), ctx);
    manager->SetStopTime(spec.simTime);
 
    auto capturing = std::make_unique<fhsim::test::CapturingObserver>();
    auto* obs      = capturing.get();
    manager->RegisterObserver(std::move(capturing));
 
    ASSERT_EQ(manager->Run(), fhsim::simRes_Completed);
 
    EXPECT_NEAR(obs->FinalOutput("spring", "Pos"), 0.0, 1e-4);
}

If the test also needs regression comparison, keep ctx.outputPath = spec.outFile.string(). The FileOutputObserver and CapturingObserver coexist without conflict.

OutputAt vs StateAt

CapturingObserver exposes two lookup methods:

// Output port value — keyed by objectName and signalName from the XML output spec.
double OutputAt(size_t step, const string& objectName, const string& signalName,
                int scalarIndex = 0);
double FinalOutput(const string& objectName, const string& signalName,
                   int scalarIndex = 0);
 
// ODE state value — keyed by objectName and tagName from AddState().
double StateAt(size_t step, const string& objectName, const string& tagName);
double FinalState(const string& objectName, const string& tagName);

Use OutputAt for most tests. The XML output spec is the declared observable interface of the model. Output names (objectName, signalName) are stable across XML reorganisation. OutputAt accesses the same values that result.output and CompareRegression use, making it consistent with the regression workflow.

Use StateAt when:

• The quantity you need is not exposed as an output port in the XML.
• You are writing a tight unit test for a new SimObject whose XML has no <Output> element yet.
• You need to verify a state at step 0 (IC validation), where output scheduling may not emit a snapshot.

StateAt is keyed by the objectName and tagName strings that the SimObject passes to ISimObjectCreator::AddState(). These are stable as long as the SimObject's state registration does not change, but they can be affected by reordering SimObjects in the XML.

Decision table

Situation	Use
XML declares an output column for this value	OutputAt / FinalOutput
Asserting on a value not in the XML output spec	StateAt / FinalState
Regression comparison across runs	RunTest + CompareRegression
Checking that the simulation completes without error	result.ok from RunTest

Convenience methods

obs->StepCount()           // number of snapshots collected (dataProduced == true)
obs->TimeAt(step)          // simulation time at step
obs->Metadata()            // full SimMetadata (state names, output columns, …)

The step index (0-based) runs over output steps only — internal integrator steps that do not produce output are not counted, matching the cadence of the CSV result file.

---

Main data structures

• TestSpec: ID, timing limits, runtime config snapshot, resolved paths, Jacobian policy, optional diagnosticsJsonPath.
• TestResult: ok/error, timing metrics, resolved paths, parsed output, Jacobian run result.
• JacobianRunResult (in TestResult::jacobian): status (NotRequested, SkippedUnavailable, Passed, Failed, Error), diagnostics, and per-scope summaries.
• PerformanceMetrics: wall/cpu/sim-time/step counters and rates.
• RegressionReport: ok + accumulated diagnostic text.
• ResultFileData / ObjectTimeSeries: parsed output columns.
• FindSeries(result, objectName, tagName): locate a specific output column quickly.
• CapturingObserver: in-memory observer with OutputAt/StateAt/FinalOutput/FinalState accessors.

Example:

const auto* state = fhsim::test::FindSeries(run.output, "MyObject", "speed_0");
ASSERT_NE(state, nullptr);
EXPECT_NEAR(state->m_colData.second.back(), expectedSpeed, 1e-6);

Optional diagnostics JSON output

Set TestSpec::diagnosticsJsonPath to persist unified diagnostics data after a successful RunTest(spec) call.

RunTest writes this file only when diagnostics are enabled in the simulation input (for example by adding <Diagnostics/> under <Integrator>).

fhsim::test::TestSpec spec;
spec.ID                  = "ImplicitIntegration_Euler1imp";
spec.runtimeConfig       = GetTestRuntimeConfig();
spec.simTime             = 1.0;
spec.diagnosticsJsonPath = "out/ImplicitIntegration_Euler1imp_diag.json";
 
const auto run = fhsim::test::RunTest(spec);
ASSERT_TRUE(run.ok) << run.error;

The file contains:

• metadata: state and port naming metadata (simObjectNames, stateNames, portObjectNames, portNames, portIndices).
• snapshots: by default the latest snapshot; full history when history accumulation is enabled by the integrator/diagnostics pipeline.

Standalone Jacobian checking

For tests that need to verify a SimObject's analytical Jacobian independently of the RunTest pipeline, use CheckJacobians or CheckJacobiansFromXml from JacobianChecker.h (included transitively by FhsimTest.h and TestSpec.h):

#include <fhsim/testtools/JacobianChecker.h>
 
fhsim::test::JacobianCheckConfig cfg;
cfg.relTol               = 1e-4;  
cfg.absTol               = 1e-10; 
cfg.perturbationEps      = 1e-8;  
cfg.randomSamples        = 3;     
cfg.maxViolationsReported = 20;
cfg.timePoints           = {0.0, 5.0}; 
 
// From file:
auto results = fhsim::test::CheckJacobians("path/to/input.xml", cfg);
 
// From in-memory XML string:
auto results2 = fhsim::test::CheckJacobiansFromXml(xmlString, cfg);
 
for (const auto& r : results)
    EXPECT_TRUE(r.ok) << r.Summary();

Each element of the returned vector covers one *(state-point × scope)* combination, where scope is "system" plus one entry per SimObject that returns HasJacobians() == true.

JacobianCheckResult field	Type	Meaning
ok	bool	All elements within tolerance
label	string	Human-readable scope label
scope	string	"system" or SimObject name
sampleIndex	int	0 = IC, >0 = random perturbation index
timeIndex	int	Index into config.timePoints
atTime	double	Simulation time of the check
nStates	int	Jacobian block dimension
maxAbsErr	double	Worst absolute error across all elements
maxRelErr	double	Worst relative error across all elements
mismatchCount	int	Number of elements exceeding tolerances
violations	vector<JacobianElementError>	Up to maxViolationsReported worst elements
error	string	Non-empty on setup/runtime failure
Summary()	—	Formatted multi-line diagnostic string

Each JacobianElementError carries row, col, analytical, numerical, absErr, and relErr.

Utility: NullLogFile

To suppress log output in tests, use the platform-portable null device helper:

#include <fhsim/testtools/NullLogFile.h>
 
// Returns "/dev/null" on Unix, "NUL" on Windows.
spec.logFile = fhsim::test::NullLogFile();

This avoids hard-coding platform-specific paths in test code.

Troubleshooting

Symptom	Likely cause	Action
Missing input file	ID-to-path mapping mismatch	Verify test ID and <inPath>/<group>/<ID>_in.xml
Missing .ref in ReadOnly mode	Baseline not created	Run once with CreateIfMissing or Create
Missing _ref.csv with output_rms_max	Output reference absent	Generate baseline output reference
Wall-time budget exceeded	maxRunTime too low or simulation regressed	Raise budget for heavy tests or investigate performance
Output identity mismatch	Changed output column names/order	Update reference and/or adapt test expectations
Jacobian check skipped in Auto	Model does not expose Jacobians	This is expected in Auto; use Yes to require Jacobians
Jacobian hard-fail in Yes mode	Jacobians unavailable or numerical/analytical mismatch	Inspect result.jacobian.diagnostics and failing scope summaries