Testing¶

Overview¶

Verification, Validation¶

Correctness of the simulation model is a primary concern of the developers and users of the model, because they want to obtain credible simulation results. Verification and validation are activities conducted during the development of a simulation model with the ultimate goal of producing an accurate and credible model.

• Verification of a model is the process of confirming that it is implemented correctly according to the conceptual model. This means that it matches the specifications and assumptions deemed acceptable for the intended purpose of application. During verification, the model is tested to identify and fix errors in its implementation.

• Validation checks the accuracy of the model in representing the real system. Model validation is defined as the “substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model”. A model should be constructed for a specific purpose or set of objectives, and its validity determined for that purpose.

Of the two, verification is essentially a software engineering issue, so it can be assisted with tools used for software quality assurance, for example testing tools. Validation, on the other hand, is not a software engineering issue.

Unit Testing, Regression Testing¶

As mentioned earlier, software testing techniques can significantly aid in model verification. Testing can also help ensure that a simulation model, once validated and verified, remains correct for an extended period.

Software testing is an independent discipline with various techniques and methodologies. Here, we will only mention two types that are relevant to us: regression testing and unit testing.

• Regression testing is a technique that aims to uncover new software bugs, or regressions, in existing areas of a system after changes such as enhancements, patches, or configuration changes have been made to them.

• Unit testing is a method in which individual units of source code are tested to determine if they are suitable for use. In an object-oriented environment, this is usually done at the class level.

The two may overlap; for example, unit tests are also useful for detecting regressions.

One way to perform regression testing on an OMNeT++ simulation model is to record the log produced during simulation and compare it to a pre-recorded log. However, code refactoring may change the log in nontrivial ways, making it impossible to compare it to the pre-recorded one. Alternatively, one can compare only the result files or certain simulation results, avoiding the effects of refactoring, but some regressions may go undetected. Such tradeoffs are typical in regression testing.

Unit testing of simulation models can be done at the class or module level. There are many open-source unit testing frameworks for C++, such as CppUnit, Boost Test, Google Test, and UnitTest++, to name a few. They are well-suited for class-level testing. However, applying them to module testing can be cumbersome due to the peculiarities of the domain (network simulation) and OMNeT++.

A test in an xUnit-type testing framework (a collective name for CppUnit-style frameworks) operates with various assertions to test function return values and object states. This approach is challenging to apply in the testing of OMNeT++ modules, which often operate in a complex environment (cannot be easily instantiated and operated in isolation), respond to various events (messages, packets, signals, etc.), and have complex dynamic behavior and substantial internal state.

Subsequent sections will introduce opp_test, a tool provided by OMNeT++ to assist with various testing tasks, and summarize various testing methods useful for testing simulation models.

The opp_test Tool¶

Introduction¶

This section documents opp_test, a versatile tool that is helpful for various testing scenarios. opp_test can be used for various types of tests, including unit tests and regression tests. It was originally written for testing the OMNeT++ simulation kernel, but it is equally suited for testing functions, classes, modules, and whole simulations.

opp_test is built around a simple concept: it allows the user to define simulations in a concise way, run them, and check that the output (result files, log, etc.) matches a predefined pattern or patterns. In many cases, this approach works better than inserting various assertions into the code (which is still also an option).

Each test is a single file with the .test file extension. All NED code, C++ code, ini files, and other data necessary to run the test case, as well as the PASS criteria, are packed together in the test file. Such self-contained tests are easier to handle and also encourage authors to write tests that are compact and to the point.

Let’s see a small test file, cMessage_properties_1.test:

%description:
Test the name and length properties of cPacket.

%activity:
cPacket *pk = new cPacket();
pk->setName("ACK");
pk->setByteLength(64);
EV << "name: " << pk->getName() << endl;
EV << "length: " << pk->getByteLength() << endl;
delete pk;

%contains: stdout
name: ACK
length: 64

What this test says is this: create a simulation with a simple module that has the above C++ code block as the body of the activity() method, and when run, it should print the text after the %contains line.

To run this test, we need a control script, for example runtest from the omnetpp/test/core directory. runtest itself relies on the opp_test tool.

The output will be similar to this one:

$ ./runtest cMessage_properties_1.test
opp_test: extracting files from *.test files into work...
Creating Makefile in omnetpp/test/core/work...
cMessage_properties_1/test.cc
Creating executable: out/gcc-debug/work
opp_test: running tests using work.exe...
*** cMessage_properties_1.test: PASS
========================================
PASS: 1   FAIL: 0   UNRESOLVED: 0

Results can be found in work/

This was a passing test. What would constitute a failure?

• crash

• simulation runtime error

• nonzero exit code (a simulation runtime error is also detected by nonzero exit code)

• the output doesn’t match the expectation (there are several possibilities for expressing what is expected: multiple match criteria, literal string vs regex, positive vs negative match, matching against the standard output, standard error, or any file, etc.)

One normally wants to run several tests together. The runtest script accepts several .test files on the command line, and when started without arguments, it defaults to *.test, all test files in the current directory. At the end of the run, the tool prints summary statistics (number of tests passed, failed, and unresolved).

An example run from omnetpp/test/core (some lines were removed from the output, and one test was changed to show a failure):

$ ./runtest cSimpleModule-*.test
opp_test: extracting files from *.test files into work...
Creating Makefile in omnetpp/test/core/work...
[...]
Creating executable: out/gcc-debug/work
opp_test: running tests using work...
*** cSimpleModule_activity_1.test: PASS
*** cSimpleModule_activity_2.test: PASS
[...]
*** cSimpleModule_handleMessage_2.test: PASS
*** cSimpleModule_initialize_1.test: PASS
*** cSimpleModule_multistageinit_1.test: PASS
*** cSimpleModule_ownershiptransfer_1.test: PASS
*** cSimpleModule_recordScalar_1.test: PASS
*** cSimpleModule_recordScalar_2.test: FAIL (test-1.sca fails %contains-regex(2) rule)
expected pattern:
>>>>run General-1-.*?
scalar Test     one     24.2
scalar Test     two     -1.5888<<<<
actual output:
>>>>version 2
run General-1-20141020-11:39:34-1200
attr configname General
attr datetime 20141020-11:39:34
attr experiment General
attr inifile _defaults.ini
[...]
scalar Test     one     24.2
scalar Test     two     -1.5
<<<<
*** cSimpleModule_recordScalar_3.test: PASS
*** cSimpleModule_scheduleAt_notowner_1.test: PASS
*** cSimpleModule_scheduleAt_notowner_2.test: PASS
[...]
========================================
PASS: 36   FAIL: 1   UNRESOLVED: 0
FAILED tests: cSimpleModule_recordScalar_2.test

Results can be found in work/

Note that code from all tests was linked to form a single executable, which saves time and disk space compared to per-test executables or libraries.

A test file like the one above is useful for unit testing of classes or functions. However, as we will see, the test framework provides further facilities that make it convenient for testing modules and whole simulations as well.

The following sections go into details about the syntax and features of .test files, about writing the control script, and give advice on how to cover several use cases with the opp_test tool.

Terminology¶

The next sections will use the following terminology:

• test file: A file with the .test extension that opp_test understands.

• test tool: The opp_test program

• control script: A script that relies on opp_test to run the tests. The control script is not part of OMNeT++ because it usually needs to be somewhat specific to the simulation model or framework being tested.

• test program: The simulation program whose output is checked by the test. It is usually work/work (work/work.exe on Windows). However, it is also possible to let the control script build a dynamic library from the test code, and then use e.g. opp_run as the test program.

• test directory: The directory where a .test file is extracted; usually work/<testname>/. It is also set as the working directory for running the test program.

Test File Syntax¶

Test files are composed of %-directives of the following syntax:

%<directive>: <value>
<body>

The body extends up to the next directive (the next line starting with %), or to the end of the file. Some directives require a value, others a body, or both.

Certain directives, e.g. , may occur several times in the file.

Test Description¶

Syntax:

%description:
<test-description-lines>

is customarily written at the top of the .test file and allows one to provide a multi-line comment about the purpose of the test. It is recommended to invest time in well-written descriptions because they make determining the original purpose of a test that has become broken significantly easier.

Test Code Generation¶

This section describes the directives used for creating C++ source and other files in the test directory.

%activity¶

Syntax:

%activity:
<body-of-activity()>

lets one write test code without much boilerplate. The directive generates a simple module that contains a single activity() method with the given code as the method body.

A NED file containing the simple module’s (bare-bones) declaration, and an ini file to set up the module as a network are also generated.

%module¶

Syntax:

%module: <modulename>
<simple-module-C++-definition>

lets one define a module class and run it as the only module in the simulation.

A NED file containing the simple module’s (bare-bones) declaration, and an ini file to set up the module as a network are also generated.

%includes, %global¶

Syntax:

%includes:
<#include directives>

%global:
<global-code-pasted-before-activity>

and are helpers for and , and let one insert additional lines into the generated C++ code.

Both directives insert the code block above the module C++ declaration. The only difference is in their relation to the C++ namespace: the body of is inserted above (i.e., outside) the namespace, and the body of is inserted inside the namespace.

The Default Ini File¶

The following ini file is always generated:

[General]
network = <network-name>
cmdenv-express-mode = false

The network name in the file is chosen to match the module generated with or ; if they are absent, it will be Test.

%network¶

Syntax:

%network: <network-name>

This directive can be used to override the network name in the default ini file.

%file, %inifile¶

Syntax:

%file: <file-name>
<file-contents>

%inifile: [<inifile-name>]
<inifile-contents>

saves a file with the given file name and content into the test’s extraction folder in the preparation phase of the test run. It is customarily used for creating NED files, MSG files, ini files, and extra data files required by the test. There can be several sections in the test file.

is similar to in that it also saves a file with the given file name and content, but it additionally also adds the file to the simulation’s command line, causing the simulation to read it as an (extra) ini file. There can be several sections in the test file.

The default ini file is always generated.

The @TESTNAME@ Macro¶

In test files, the string @TESTNAME@ will be replaced with the test case name. Since it is substituted everywhere (C++, NED, msg, and ini files), one can also write things like @TESTNAME@_function() or printf("this is @TESTNAME@\n").

Avoiding C++ Name Clashes¶

Since all sources are compiled into a single test executable, actions have to be taken to prevent accidental name clashes between C++ symbols in different test cases. A good way to ensure this is to place all code into namespaces named after the test cases.

namespace @TESTNAME@ {
   ...
};

This is done automatically for the , , and blocks, but for other files (e.g., source files generated via ), that needs to be done manually.

PASS Criteria¶

%contains, %contains-regex, %not-contains, %not-contains-regex¶

Syntax:

%contains: <output-file-to-check>
<multi-line-text>

%contains-regex: <output-file-to-check>
<multi-line-regexp>

%not-contains: <output-file-to-check>
<multi-line-text>

%not-contains-regex: <output-file-to-check>
<multi-line-regexp>

These directives let one check for the presence (or absence) of certain text in the output. One can check a file, or the standard output or standard error of the test program; for the latter two, stdout and stderr need to be specified as file names, respectively. If the file is not found, the test will be marked as an error. There can be several -style directives in the test file.

The text or regular expression can be multi-line. Before matching is attempted, trailing spaces are removed from all lines in both the pattern and the file contents; leading and trailing blank lines in the patterns are removed; and any substitutions are performed (see ). Perl-style regular expressions are accepted.

To facilitate debugging of tests, the text/regex blocks are saved into the test directory.

%subst¶

Syntax:

%subst: /<search-regex>/<replacement>/<flags>

It is possible to apply text substitutions to the output before it is matched against expected output. This is done with the directive; there can be more than one in a test file. It takes a Perl-style regular expression to search for, a replacement text, and flags, in the /search/replace/flags syntax. Flags can be empty or a combination of the letters i, m, and s, for case-insensitive, multi-line, or single-string match (see the Perl regex documentation.)

was primarily invented to deal with differences in printf output across platforms and compilers: different compilers print infinite and not-a-number in different ways: 1.#INF, inf, Inf, -1.#IND, nan, NaN, etc. With , they can be brought to a common form:

%subst: /-?1\.#INF/inf/
%subst: /-?1\.#IND/nan/
%subst: /-?1\.#QNAN/nan/
%subst: /-?NaN/nan/
%subst: /-?nan/nan/

%round-numbers¶

Syntax:

%round-numbers: <significant-digits>

This directive rounds all floating-point numbers in the output to the specified number of significant digits before matching against expected output. It is applied before rules. The rounding uses %.<N>g formatting, which strips trailing zeros.

This is useful for eliminating floating-point representation noise that may appear when output precision is high. For example, a bin edge that is mathematically 0.06 may be stored as the IEEE 754 double 0.059999999999999998. At high output precision, this noise becomes visible, and poses a challenge when specifying the expected output. With , the value is first properly rounded, so subsequent truncation produces the correct result.

%exitcode, %ignore-exitcode¶

Syntax:

%exitcode: <one-or-more-numeric-exit-codes>

%ignore-exitcode: 1

and let one test the exit code of the test program. The former checks that the exit code is one of the numbers specified in the directive; the latter makes the test framework ignore the exit code.

OMNeT++ simulations exit with zero if the simulation terminated without an error, and some >0 code if a runtime error occurred. Normally, a nonzero exit code makes the test fail. However, if the expected outcome is a runtime error (e.g., for some negative test cases), one can use either to express that or specify and test for the presence of the correct error message in the output.

%file-exists, %file-not-exists¶

Syntax:

%file-exists: <filename>

%file-not-exists: <filename>

These directives test for the presence or absence of a certain file in the test directory.

Extra Processing Steps¶

%env, %extraargs, %testprog¶

Syntax:

%env: <environment-variable-name>=<value>

%extraargs: <argument-list>

%testprog: <executable>

The directive lets one set an environment variable that will be defined when the test program and the potential pre- and post-processing commands run. There can be multiple directives in the test file.

lets one add extra command-line arguments to the test program (usually the simulation) when it is run.

The directive lets one replace the test program. also slightly alters the arguments the test program is run with. Normally, the test program is launched with the following command line:

$ <default-testprog> -u Cmdenv <test-extraargs> <global-extraargs> <inifiles>

When is present, it becomes the following:

$ <custom-testprog> <test-extraargs> <global-extraargs>

That is, -u Cmdenv and <inifilenames> are removed; this allows one to invoke programs that do not require or understand them and puts the test author in complete command of the arguments list.

Note that and have an equivalent command-line option in opp_test. (In the text above, <global-extraargs> stands for extra args specified to opp_test.) doesn’t need an option in opp_test because the test program inherits the environment variables from opp_test, so one can just set them in the control script or in the shell one runs the tests from.

%prerun-command, %postrun-command¶

Syntax:

%prerun-command: <command>

%postrun-command: <command>

These directives let one run extra commands before/after running the test program (i.e., the simulation). There can be multiple pre- and post-run commands. The post-run command is useful when the test outcome cannot be determined by simple text matching but requires statistical evaluation or other processing.

If the command returns a nonzero exit code, the test framework will assume that it is due to a technical problem (as opposed to test failure) and count the test as an error. To make the test fail, let the command write a file and match the file’s contents using & co.

If the post-processing command is a short script, it is practical to add it into the .test file via the directive, and invoke it via its interpreter. For example:

%postrun-command: python test.py
%file: test.py
<Python script>

Or:

%postrun-command: R CMD BATCH test.R
%file: test.R
<R script>

If the script is very large or shared among several tests, it is more practical to place it into a separate file. The test command can find the script e.g., by a relative path or by referring to an environment variable that contains its location or full path.

Error¶

A test case is considered to be in error if the test program cannot be executed at all, the output cannot be read, or some other technical problem occurred.

Expected Failure¶

can be used in the test file to force a test case to ignore a failure. If a test case marked with %expected-failure fails, it will be counted as expectfail instead of fail. opp_test will return successfully if no test cases reported fail or error results.

Syntax:

%expected-failure: <single-line-reason-for-allowing-a-failure>

Skipped¶

A test case can be skipped if the current system configuration does not allow its execution (e.g., certain optional features are not present). Skipping is done by printing #SKIPPED or #SKIPPED:some-explanation on the standard output, at the beginning of the line.

opp_test Synopsis¶

Little has been said so far about what opp_test actually does or how it is meant to be run. opp_test has two modes: file generation and test running. When running a test suite, opp_test is actually run twice, once in file generation mode and then in test running mode.

File generation mode has the syntax . For example:

$ opp_test gen *.test

This command will extract C++ and NED files, ini files, etc., from the .test files into separate files. All files will be created in a work directory (which defaults to ./work/), and each test will have its own subdirectory under ./work/.

The second mode, test running, is invoked as . For example:

$ opp_test run *.test

In this mode, opp_test will run the simulations, check the results, and report the number of passes and failures. The way of invoking simulations (which executable to run, the list of command-line arguments to pass, etc.) can be specified to opp_test via command-line options.

The simulation needs to have been built from source before can be issued. Usually, one would employ a command similar to

$ cd work; opp_makemake --deep; make

to achieve that.

Writing the Control Script¶

Usually, one writes a control script to automate the two invocations of opp_test and the build of the simulation model between them.

A basic variant would look like this:

#! /bin/sh
opp_test gen -v *.test || exit 1
(cd work; opp_makemake -f --deep; make) || exit 1
opp_test run -v *.test

For any practical use, the test suite needs to refer to the codebase being tested. This means that the codebase must be added to the include path, must be linked with, and the NED files must be added to the NED path. The first two can be achieved by the appropriate parameterization of opp_makemake, and the last one can be done by setting and exporting the NEDPATH environment variable in the control script.

For inspiration, check out runtest in the omnetpp/test/core directory and a similar script used in the INET Framework.

* * *

Further sections describe how one can implement various types of tests in OMNeT++.

Smoke Tests¶

Smoke tests are a tool for basic verification and regression testing. Essentially, the simulation is run for a period of time without crashing or experiencing any runtime errors. Naturally, smoke test provide very low confidence in the model, but in turn they are simple to implement.

Automation is crucial. The INET Framework includes a script that can run all or selected simulations defined in a CSV file, which includes columns such as the working directory and the command to run. The script also generates a report of the results. It can easily be customized for other models or model frameworks.

Fingerprint Tests¶

Fingerprint tests are a low-cost but effective tool for regression testing simulation models. A fingerprint is a hash computed from various properties of simulation events, messages, and statistics. The hash value is continuously updated as the simulation executes, and thus, the final fingerprint value is a characteristic of the simulation’s trajectory. For regression testing, one needs to compare the computed fingerprints to those from a reference run – if they differ, the simulation trajectory has changed. In general, fingerprint tests are very useful for ensuring that a change (such as refactoring, bugfix, or a new feature) hasn’t broken the simulation.

Fingerprint Computation¶

Technically, providing a fingerprint option in the config file or on the command line (-­fingerprint=...) will enable fingerprint computation in the OMNeT++ simulation kernel. When the simulation terminates, OMNeT++ compares the computed fingerprints with the provided ones, and if they differ, an error is generated.

Ingredients¶

The fingerprint computation algorithm allows controlling what is included in the hash value. Changing the ingredients allows one to make the fingerprint sensitive to certain changes while keeping it immune to others.

The ingredients of a fingerprint are usually indicated after a / sign following the hexadecimal hash value. Each ingredient is identified with a letter. For example, t stands for simulation time. Thus, the following omnetpp.ini line

fingerprint = 53de-64a7/tplx

means that a fingerprint needs to be computed with the simulation time, the module’s full path, received packet’s bit length, the extra data included for each event, and the result should be 53de-64a7.

Ingredients may also be specified with the fingerprint-ingredients configuration option. However, that is rarely necessary because the ingredients list included in the fingerprints takes precedence and is also more convenient to use.

The full list of fingerprint ingredients:

Most fingerprint ingredients require no explanation, but the ones marked with asterisk above do:

• message contents: Incorporates the core message data, excluding control info and technical bookkeeping fields from the last transmission. This ingredient is insensitive to simulation time and the identity of sender/receiver modules. Uses parsimPack() to serialize objects into byte arrays for fingerprint inclusion.

• message control info: Includes control information objects attached to arriving messages. Uses parsimPack() to serialize objects into byte arrays for fingerprint inclusion.

• message data: Legacy ingredient for message data inclusion. Uses parsimPack() to serialize objects into byte arrays for fingerprint inclusion. This is retained for backward compatibility with older fingerprints.

• extra data added programmatically: Includes data explicitly added to the fingerprint via addExtraData() method calls in the simulation code. This allows custom data to be incorporated into fingerprint computation.

• canvas figures: Includes information about graphical figures displayed on the simulation canvas. Primarily incorporates geometry information (positions, sizes, shapes) but excludes styling details (colors, fonts, line styles). This design choice makes fingerprints more practical for regression testing, as geometry typically carries important functional information while styling is often a matter of visual preference.

These ingredients enable more detailed fingerprint-based regression tests and can help detect subtle differences in message traffic and simulation behavior that might not be captured by other fingerprint ingredients.

Multiple Fingerprints, Alternative Values¶

It is possible to specify more than one fingerprint, separated by commas, each with different ingredients. This will cause OMNeT++ to compute multiple fingerprints, and all of them must match for the test to pass. For example:

fingerprint = 53de-64a7/tplx, 9a3f-7ed2/szv

Occasionally, the same simulation gives a different fingerprint when run on a different processor architecture or platform. This is due to subtle differences in floating-point arithmetic across platforms. [1] Acknowledging this fact, OMNeT++ lets one list several values for a fingerprint, separated by spaces, and will accept whichever is produced by the simulation. The following example lists two alternative values for both fingerprints.

fingerprint = 53de-64a7/tplx 63dc-ff21/tplx, 9a3f-7ed2/szv da39-91fc/szv

Note that fingerprint computation has been changed and significantly extended in OMNeT++ version 5.0. [2]

Further Filtering¶

It is also possible to filter which modules, statistics, etc. are included in the fingerprints. The fingerprint-events, fingerprint-modules, and fingerprint-results options filter events, modules, and statistical results, respectively. These options take wildcard expressions that are matched against the corresponding object before including its property in the fingerprint. These filters are mainly useful to limit fingerprint computation to certain parts of the simulation.

Programmatic Access¶

cFingerprintCalculator is the class responsible for fingerprint computation. The current fingerprint computation object can be retrieved from cSimulation using the getFingerprintCalculator() member function. This method will return nullptr if fingerprint computation is turned off for the current simulation run.

To contribute data to the fingerprint, cFingerprintCalculator has several addExtraData() methods for various data types (string, long, double, byte array, etc.)

An example (note that we check the pointer for nullptr to decide whether a fingerprint is being computed):

cFingerprintCalculator *fingerprint = getSimulation()->getFingerprintCalculator();
if (fingerprint) {
    fingerprint->addExtraData(retryCount);
    fingerprint->addExtraData(rttEstimate);
}

Data added using addExtraData() will only be counted in the fingerprint if the list of fingerprint ingredients contains x (otherwise addExtraData() does nothing).

Fingerprint Tests¶

The INET Framework also contains a script for automated fingerprint tests. The script runs all or selected simulations defined in a CSV file (with columns like the working directory, the command to run, the simulation time limit, and the expected fingerprints), and reports the results. The script is extensively used during INET Framework development to detect regressions and can be easily adapted to other models or model frameworks.

Excerpt from a CSV file that prescribes fingerprint tests to run:

examples/aodv/, ./run -f omnetpp.ini -c Static,   50s,  4c29-95ef/tplx
examples/aodv/, ./run -f omnetpp.ini -c Dynamic,  60s,  8915-f239/tplx
examples/dhcp/, ./run -f omnetpp.ini -c Wired,    800s, e88f-fee0/tplx
examples/dhcp/, ./run -f omnetpp.ini -c Wireless, 500s, faa5-4111/tplx

Unit Tests¶

If a simulation model contains units of code (classes, functions) smaller than a module, they are candidates for unit testing. For a network simulation model, examples of such classes are network addresses, fragmentation reassembly buffers, queues, various caches and tables, serializers and deserializers, checksum computation, etc.

Unit tests can be implemented as .test files using the opp_test tool (the directive is especially useful here), or with potentially any other C++ unit testing framework.

When using .test files, the build part of the control script needs to be set up so that it adds the tested library’s source folder(s) to the include path, and also links the library to the test code.

Module Tests¶

OMNeT++ modules are not as easy to unit test as standalone classes because they typically assume a more complex environment, and, especially modules that implement network protocols, participate in more complex interactions than the latter.

To test a module in isolation, one needs to place it into a simulation where the module’s normal operation environment (i.e., other modules it normally communicates with) is replaced by mock objects. Mock objects are responsible for providing stimuli for the module under test and (partly) for checking the response.

Module tests may be implemented in .test files using the opp_test tool. A .test file allows one to place the test description, test setup, and expected output into a single, compact file, while large files or files shared among several tests may be factored out and only referenced by .test files.

Statistical Tests¶

Statistical tests are tests where the outcome is determined based on statistical properties of the simulation results.

Statistical tests can be used for validation as well as regression testing.

Validation Tests¶

Validation tests aim to verify that simulation results correspond to some reference values, ideally those obtained from the real system. In practice, reference values can come from physical measurements, theoretical values, or results from another simulator.

Statistical Regression Tests¶

After refactoring that changes the simulation trajectory (e.g. eliminating or introducing extra events, or changes in RNG usage), there may be no other way to perform regression testing than to check that the model produces statistically the same results as before.

For statistical regression tests, one needs to run multiple simulations with the same configuration but different RNG seeds, and verify that the results follow the same distributions as before. Student’s t-test (for mean) and the F-test (for variance) can be used to check if the “before” and the “after” result sets are from the same distribution.

Implementation¶

Statistical software like GNU R is extremely useful for performing these tests.

Statistical tests can also be implemented in .test files. To run several simulations within one test, use to pass the option to Cmdenv; alternatively, can be used to have the test tool run opp_runall instead of the normal simulation program. For performing statistical computations, use to run a Python or R script. The Python script may rely on the Pandas, SciPy, and omnetpp.scave packages for reading and evaluating OMNeT++ result files.

The INET Framework contains statistical tests that can provide inspiration.