Skip to content

Device testing workflow

Jump to bottom
Francesco Anselmo edited this page Aug 20, 2019 · 8 revisions

In this tutorial you will go through the workflow to test a physical device with DAQ, generating test reports that can be stored in a repository supporting revision history.

You will:

  1. Set up DAQ in test lab mode,
  2. Set up the reporting template,
  3. Edit a file to describe the physical device,
  4. Select the tests to be run for the device,
  5. Execute the tests,
  6. Edit the report that has been output to include summary notes,
  7. Store the report in a repository supporting revision history.
  8. Convert the report from markdown to PDF so that it can be distributed and stored on a different system.

Prerequisites

The prerequisites for this tutorial are:

  1. a good understanding of how to set up and run DAQ
  2. having set up the DAQ dashboard (optional)

Set up DAQ in test lab mode

To set up DAQ in test lab mode, follow the tutorial Using DAQ with an OpenFlow switch.

Set up reporting

Reporting output in DAQ makes use of a report template.

An example of the report template is stored in qualification/report_template.md

To use the report template, copy the report template to the local/site/ folder:

cp qualification/report_template.md local/site/

Edit the physical device description

To set up the physical device description, do a first DAQ dry run to make it generate the device folder.

  1. connect device to the DAQ system, plugging it to a switch port
  2. run DAQ in single shot mode: cmd/run -s
  3. check that local/site/mac_addr/############ has been created, where ############ is the MAC address of the connected device

After the local/site/mac_addr/############ has been generate, copy the relevant configuration JSON template files in the local/site and local/site/mac_addr/############ folders.

  1. cp qualification/system_module_config.json local/site/module_config.json
  2. cp qualification/device_module_config.json local/site/mac_addrs/############/module_config.json where ############ is the MAC address of the connected device being tested

Now edit the local/site/mac_addrs/############/module_config.json file, including in the respective fields relevant information about the device. For instance, substitute all the fields with asterisks (*) with relevant information:

...
  "device_datasheets_url": "*** Device Datasheets URL ***",
  "device_description": "*** Device Description ***",
  "device_image": "*** Device Image URL ***",
  "device_info": {
    "guid": "*** GUID ***",
    "hostname": "*** Network Hostname ***",
    "make": "*** Make ***",
    "model": "*** Model ***",
    "name": "*** Name ***",
    "serial": "*** Serial ***",
    "type": "*** Type ***",
    "firmware_version": "*** Firmware Version ***"
  },
  "device_manuals_url": "*** Device Manuals URL ***",
  "device_type": "*** Device Type ***",
...

Note that the device_type field refers to the name of a folder placed inside the local/site/device_types directory where the inclusion of a module_config.json file with pre-populated device fields automatically fills in the other device information fields. It can therefore act as a shortcut to avoid typing this information for devices with the same make and model.

For instance, instead of editing each device independently, place a module_config.json file inside local/site/device_types/type_of_device, where type_of_device is your arbitrary name for a device type. Add a "device_type": "type_of_device" line inside your local/site/mac_addrs/############/module_config.json` you can then get this configuration replicated across similar devices.

Select the tests

To include all the tests to the DAQ execution, add the line host_tests=misc/all_tests.conf to local/system.conf.

If these tests have not been built yet, run cmd/build from the daq folder.

Edit the local/site/module_config.json and local/site/mac_addrs/############/module_config.json according to your needs (enable/disable test module, edit operator name, etc.).

The local/site/module_config.json includes test configuration information for quality assurance in the "process", "report" and "tests" JSON objects.

It also includes a "modules" JSON object to enable or disable test modules at a system level.

The local/site/mac_addrs/############/module_config.json that has been edited in the previous step also has a "modules" JSON object to enable or disable test modules at a device level. It also includes a "servers" JSON object to specify which ports are expected to be enabled or disabled on the device. This will need to be edited according to the expected project/device requirements.

All these files can also be edited in the web user interface by clicking on the DAQ site name for system wide settings and on the device MAC address for device specific settings.

TODO: add snapshots of web user interface.

Tests execution

The tests can now be executed in single shot mode:

cmd/run -s

or in continuous testing mode:

cmd/run

Optionally, if the web user interface has been set up, DAQ can be run in "no test mode" with the following command:

cmd/run -n

This will initialise all the connected devices and wait for the tests to be triggered by clicking on the purple button/table cell next to the switch port number.

Report output

Once the tests have completed, check inside inst/reports to get the device report (for example inst/reports/report_9a02571e8f00_2019-06-24T115221+0000.md).

The last report is also saved in the local/site/mac_addrs/############/ folder.

Optionally, if the web user interface has been set up, the report is linked in the last column of the DAQ testing table.

Report editing

The report is saved as a markdown file that can be further edited to include additional summary notes.

Upload the report to a git repository

Optionally, the report can be uploaded to a code repository that supports revision history.

Report conversion to PDF format

It is possible to convert the report from markdown to PDF so that it can be distributed and stored on a different system.

TODO

Clone this wiki locally