Examining the Code¶
This page describes in detail the contents of each used code file.
There are a few things that both the workers and the client need. These are
factored out into a common module,
utils.py. You can download the entire file
In the first half of it, there are the imports it will use, and a
helper class. We use this in place of its superclass, the built-in
It only changes the string representation of the base type, to make it shorter
than the actual contents. There is a technical reason for using this: It reduces
the amount of data transferred to, and stored in, the Redis database.
There are also two functions in it to handle ZIP archives in memory. The first is for extracting, and the second is for compressing, with the option to exclude some directories.
And the code for the jobs, worker.py:
With the actual job function:
The comments make its operation pretty straightforward.
The model needs to be cleaned, then rebuilt inside the container, because the version of some basic system libraries might not match that of those present on the host system, which would lead to incompatibility problems, possibly preventing the simulation from starting.
We select the base image to be
ubuntu:16.04, then we install Python, pip, the
dependencies of OMNeT++, and wget.
We upgrade pip using itself, then install RQ with it. It will also install the Redis client module as a dependency. Then a few environment variables need to be set, to make RQ use the right character encoding.
Next, we copy the worker source code into the image, and set the working directory.
Downloading the OMNeT++ 5.1.1 Core release archive from the official website,
extracting it, then deleting it. The referer URL has to be passed to
otherwise the server denies access. The
--progress flag is there just to
reduce the amount of textual output, which would overly pollute the build log.
bin directory added to the
PATH environment variable (which would be
setenv normally). Finally the standard building procedure is performed
make. Both graphical runtime environments and the
support for 3D rendering are disabled. The
-j $(nproc) arguments to
enable it to use all your local CPU cores when building OMNeT++ itself.
ccache to make subsequent builds of the same model sources faster:
And finally setting up the entry point to launch the rq worker, asking it to keep the results only for one minute. This will be enough, because the client will start downloading them right away, and it will reduce the amount of data stored in the Redis database on average.
Later, when we run containers from the image, we will be able to append additional arguments to the entrypoint.
The following file, client.py, implements the command-line application for submitting jobs and getting the results.
First, the usual imports:
Then a helper function to resolve the run filter in a given configuration by
opp_run tool locally, in "query" mode. This is not strictly
necessary, since using a run filter is optional, but it's a nice addition.
Defining the arguments of the tool and parsing their values:
Setting up the connection to the job queue, then using the helper function to get the actual list of run numbers. Finally pack the model source into a compressed archive (ZIP) in memory. A few directories are excluded from this archive, because they are not needed by the workers, and are usually very large.
Submitting a job into the queue for each run, storing the jobs in a list.
The run number is also written into the
meta field of each job, but that is
necessary only so we know later which run was performed by a particular job, and
we can print it when it is done. The job function itself doesn't use this, only
And finally poll for the results of each job, downloading and unpacking the output (the results) of completed jobs, and removing them from the list:
And exit when all jobs are completed.