-
Notifications
You must be signed in to change notification settings - Fork 17
codes workloads
codes-workload is an in-development abstraction layer for feeding I/O / network workloads into a simulation. It supports multiple back-ends for generating I/O and network events; data could come from a trace file, from Darshan, or from a synthetic description.
The workload generator is currently a work in progress, and the API is subject to change. The I/O generation component exposes a "POSIX-ish" open/close/read/write interface, while the network generation component exposes an "MPI-ish" send/recv/barrier/collective interface.
As an additional utility, we provide a simple debug program, src/workload/codes-workload-dump, that processes a given workload and prints to standard out.
For more details on I/O workloads, see CODES I/O workloads. For more details on how to replay application traces on network models, see CODES DUMPI workloads. For more details on replaying synthetic traffic on network models, see CODES synthetic traffic.
The codes-jobmap API (codes/codes-jobmap.h) specifies mechanisms to initialize and run multiple jobs with unique job IDs and linear namespaces, used to allow multiple concurrent workloads to be run within the same simulation. Example usage can be seen in tests/jobmap-test.c and, for the list allocation method, in tests/conf/jobmap-test-list.conf. Note that only the DUMPI trace generator has been tested using this interface at this time.
LP-IO is a set of simple reverse-computation-aware routines for conditionally outputting data on a per-LP basis. As the focus is on convenient, small-scale data output, data written via LP-IO remains in memory until the end of the simulation, or freed upon reverse computation. Large-scale, reverse-computation-aware IO is a feature we're thinking about for future usage.
The API can be found at codes/lp-io.h and is fairly self-explanatory.
The configurator is a set of scripts intended to make the auto-generation of multiple CODES configuration files easier, for the purposes of performing parameter sweeps of simulations. The configuration file defining the parameters in the parameter sweep is defined by a python source file with well-defined field names, to maximize flexibility and enable some essential features for flexible parameter sweeps (disabling certain combinations of parameters, deriving parameters from other parameters in the sweep). The actual replacement is driven by token replacement defined by the values in the configuration file.
An exhaustive example can be found at scripts/example. The scripts themselves are codes_configurator.py, codes_filter_configs.py, and codes_config_get_vals.py, each with detailed usage info. These scripts have heavily-overlapping functionality, so in the future these may be merged.