Skip to content

Using software TPG in emulated mode

Jump to bottom
Alessandro Thea edited this page Oct 1, 2024 · 1 revision

10-May-2023 - Work in progress (Adam Abed Abud)

The following instructions are for creating a software area with software tpg (SWTPG) enabled and running in emulated mode.

Instructions

Steps for creating a software area as described in here:

cd <directory_above_where_you_want_the_new_software_area>
source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt dunedaq-v4.0.0
dbt-create dunedaq-v4.0.0 <work_dir>
cd <work_dir>
cd sourcecode
git clone -b aabedabu/perf_evaluation_testing https://github.com/DUNE-DAQ/fdreadoutlibs

Create a hardware map file (e.g. hw_map.txt):

# DRO_SourceID DetLink DetSlot DetCrate DetID DRO_Host DRO_Card DRO_SLR DRO_Link 
0 0 0 1 3 localhost 0 0 0
1 1 0 1 3 localhost 0 0 1
2 0 1 1 3 localhost 0 0 2
3 1 1 1 3 localhost 0 0 3
4 0 2 1 3 localhost 0 0 4
5 1 2 1 3 localhost 0 1 0
6 0 3 1 3 localhost 0 1 1
7 1 3 1 3 localhost 0 1 2
8 0 4 1 3 localhost 0 1 3
9 1 4 1 3 localhost 0 1 4

Prepare a daqconf.json like the one provided below:

{
  "boot": {
    "use_connectivity_service": false,
    "disable_trace": false,
    "opmon_impl": "cern",
    "ers_impl": "cern",
    "use_k8s": false,
    "op_env": "swtpg_test",
    "start_connectivity_service": false
  }, 
  "readout": {
    "clock_speed_hz": 62500000,
    "data_rate_slowdown_factor": 1,
    "emulator_mode": true,
    "enable_software_tpg": true,
    "software_tpg_threshold": 1000,
    "software_tpg_algorithm": "SWTPG",  
    "thread_pinning_file": "PATH/cpupin_files/cpupin-np04-srv-028_local.json",
    "default_data_file": "asset://?label=DuneWIB&subsystem=readout",    
    "numa_config": {
            "default_id": 1,
            "default_latency_numa_aware": true,
            "exceptions": [
                {
                    "host": "np04-srv-028",
                    "card": 0,
                    "numa_id": 0
                }
            ]
        }
  },
  "trigger": {
    "trigger_window_before_ticks": 1000,
    "trigger_window_after_ticks": 1000,
    "trigger_rate_hz": 0.000001, 
    "host_trigger": "np04-srv-019.cern.ch"
  },
  "hsi": {
    "host_fake_hsi": "np04-srv-019.cern.ch",
    "host_timing_hsi": "np04-srv-019.cern.ch"
  },
  "dataflow" : {
  "host_dfo": "np04-srv-019.cern.ch",
  "apps": [
      {
        "app_name": "dataflow0",
        "token_count": 10,
        "output_paths": [ "/nfs/sw/work_dirs/aabedabu/configs_dunedaq_v4.0.0/" ],
        "host_df": "np04-srv-019.cern.ch",
        "max_file_size": 4294967296,
        "max_trigger_record_window": 0
      }
  ]
  }
}

Notes:

  • In the above file dataflow, dfo, fakehsi and trigger are hosted on np04-srv-019
  • Make sure you have the correct CPU pinning file and add the path to the file in the readout section thread_pinning_file. A sample of pinning files are available in the readoutlibs package under config/cpupins. Use the pinning files ending with "_local.json" when running in emulated mode
  • Adapt the dataflow output paths to your preferred paths

Generate a configuration from the above files. This has to be executed from the machine from which you want to run the SWTPG (e.g. np04-srv-028):

(dbt) [user@np04-srv-028 /home/]$ daqconf_multiru_gen -c ./daqconf.json --hardware-map-file hw_map.txt 10_link_swtpg

Run your partition

nanorc 10_link_swtpg/ swtpg-testing boot conf wait 60 start_run <run number> wait 180 stop_run scrap terminate

Grafana

  • Use the Intel PCM dashboard to extract the CSV information for the relevant quantities (download series joined by time)
  • Use opmon-export.py for the plotting

Example of comparison between two servers image

Clone this wiki locally