NEWS

Version 4.1.0+dev
--------------------------------------------------------------------------------

New major features:

New minor features:

Bugfixes:

Version 4.0.0
--------------------------------------------------------------------------------

New major features:

- Add PMDK backend (Adrian Jackson)
- Add CHFS backend (Osamu Tatebe)
- Add AIO backend (Julian Kunkel)
- New S3 backend (Julian Kunkel)
- New md-workbench benchmark
- New BFS backend (Osamu Tatebe)
- Add GPUDirect Storage support (Julian Kunkel)

New minor features:

- Updated DAOS backend to match latest DAOS APIs (Mohamad Chaarawi)
- Updated HDF5 backend (Jordan Henderson)
- Improved Spectrum Scale hint support (Pidad D'Souza)
- Add CSV output (Sebastian Oeste)
- Improved Gfarm support (Osamu Tatebe)
- Updated HDFS backend
- MDTest now has a directory renaming test
- Add prologue/epilogue support (GUAN Xin)
- Add stronger randomized data using golden primes (Glenn K. Lockwood)
- Add dummy backend (Julian Kunkel)
- Improved CephFS support (Pablo Llopis)
- Improved PnetCDF support (Wei-keng Liao)

Bugfixes:

- Expansive fixes to mdtest correctness (Julian Kunkel)

Version 3.3.0
--------------------------------------------------------------------------------

New major features:

- Add CephFS AIORI (Mark Nelson)
- Add Gfarm AIORI (Osamu Tatebe)
- Add DAOS AIORI (Mohamad Chaarawi)
- Add DAOS DFS AIORI (Mohamad Chaarawi)
- -B option has been replaced with --posix.odirect

New minor features:

- Display outlier host names (Jean-Yves Vet)
- Enable global default dir layout for subdirs in Lustre (Petros Koutoupis)
- Removed pound signs (#) from mdtest output file names (Julian Kunkel)
- Print I/O hints from NCMPI (Wei-keng Liao)
- Add mknod support to mdtest (Gu Zheng)
- Refactor AIORI-specific options (Julian Kunkel)
- Enable IME native backend for mdtest (Jean-Yves Vet)
- Enable mkdir/rmdir to IME AIORI (Jean-Yves Vet)
- Add HDF5 collective metadata option (Rob Latham)
- Add support for sync to AIORIs (Julian Kunkel)

General user improvements and bug fixes:

- Allocate aligned buffers to support DirectIO for BeeGFS (Sven Breuner)
- Added IOPS and latency results to json output (Robert LeBlanc)
- Fixed case where numTasks is not evenly divisible by tasksPerNode (J. Schwartz)
- Fix several memory leaks and buffer alignment problems (J. Schwartz, Axel Huebl, Sylvain Didelot)
- Add mdtest data verification (Julian Kunkel)
- Clean up functionality of stonewall (Julian Kunkel)
- Fix checks for lustre_user.h (Andreas Dilger)
- Make write verification work without read test (Jean-Yves Vet)
- Documentation updates (Vaclav Hapla, Glenn Lockwood)
- Add more debugging support (J. Schwartz)

General developer improvements:

- Fix type casting errors (Vaclav Hapla)
- Add basic test infrastructure (Julian Kunkel, Glenn Lockwood)
- Conform to strict C99 (Glenn Lockwood)

Known issues:

- S3 and HDFS backends may not compile with new versions of respective libraries

Version 3.2.1
--------------------------------------------------------------------------------

- Fixed a memory protection bug in mdtest (Julian Kunkel)
- Fixed correctness bugs in mdtest leaf-only mode (#147) (rmn1)
- Fixed bug where mdtest attempted to stat uncreated files (Julian Kunkel)

Version 3.2.0
--------------------------------------------------------------------------------

New major features:

- mdtest now included as a frontend for the IOR aiori backend (Nathan Hjelm,
  LANL)
- Added mmap AIORI (Li Dongyang, DDN)
- Added RADOS AIORI (Shane Snyder, ANL)
- Added IME AIORI (Jean-Yves Vet, DDN)
- Added stonewalling for mdtest (Julian Kunkel, U Reading)

New minor features:

- Dropped support for PLFS AIORI (John Bent)
- Added stoneWallingWearOut functionality to allow stonewalling to ensure that
  each MPI rank writes the same amount of data and captures the effects of slow
  processes (Julian Kunkel, DKRZ)
- Added support for JSON output (Enno Zickler, U Hamburg; Julian Kunkel, U
  Reading)
- Added dummy AIORI (Julian Kunkel, U Reading)
- Added support for HDF5 collective metadata operations (Rob Latham, ANL)

General user improvements:

- BeeGFS parameter support (Oliver Steffen, ThinkParQ)
- Semantics of `-R` now compares to expected signature (`-G`) (Julian Kunkel,
  DKRZ)
- Improved macOS support for ncmpi (Vinson Leung)
- Added more complete documentation (Enno Zickler, U Hamburg)
- Assorted bugfixes and code refactoring (Adam Moody, LLNL; Julian Kunkel, U
  Reading; Enno Zickler, U Hamburg; Nathan Hjelm, LANL; Rob Latham, ANL;
  Jean-Yves Vet, DDN)
- More robust support for non-POSIX-backed AIORIs (Shane Snyder, ANL)

General developer improvements:

- Improvements to build process (Nathan Hjelm, LANL; Ian Kirker, UCL)
- Added continuous integration support (Enno Zickler, U Hamburg)
- Added better support for automated testing (Julian Kunkel, U Reading)
- Rewritten option handling to improve support for backend-specific options
  (Julian Kunkel, U Reading)

Known issues:

- `api=RADOS` cannot work correctly if specified in a config file (via `-f`)
  because `-u`/`-c`/`-p` cannot be specified (issue #98)
- `writeCheck` cannot be enabled for write-only tests using some AIORIs such as
  MPI-IO (pull request #89)


Version 3.0.2
--------------------------------------------------------------------------------
- IOR and mdtest now share a common codebase. This will make it easier
  run performance benchmarks on new hardware.
- Note: this version was never properly released

Version 3.0.0
--------------------------------------------------------------------------------
- Reorganization of the build system.  Now uses autoconf/automake.  N.B. Windows
  support is not included.  Patches welcome.
- Much code refactoring.
- Final summary table is printed after all tests have finished.
- Error messages significantly improved.
- Drop all "undocumented changes".  If they are worth having, they need to be
  implemented well and documented.

Version 2.10.3
--------------------------------------------------------------------------------
- bug 2962326 "Segmentation Fault When Summarizing Results" fixed
- bug 2786285 "-Wrong number of parameters to function H5Dcreate" fixed
    (NOTE: to compile for HDF5 1.6 libs use "-D H5_USE_16_API")
- bug 1992514 "delay (-d) doesn't work" fixed

Contributed by demyn@users.sourceforge.net
- Ported to Windows. Required changes related to 'long' types, which on Windows
  are always 32-bits, even on 64-bit systems. Missing system headers and
  functions account for most of the remaining changes.
  New files for Windows:
   - IOR/ior.vcproj                - Visual C project file
   - IOR/src/C/win/getopt.{h,c}    - GNU getopt() support
  See updates in the USER_GUIDE for build instructions on Windows.
- Fixed bug in incrementing transferCount
- Fixed bugs in SummarizeResults with mismatched format specifiers
- Fixed inconsistencies between option names, -h output, and the USER_GUIDE.

Version 2.10.2
--------------------------------------------------------------------------------
- Extend existing random I/O capabilities and enhance performance
  output statistics. (Hodson, 8/18/2008)

Version 2.10.1
--------------------------------------------------------------------------------
- Added '-J' setAlignment option for HDF5 alignment in bytes; default value
  is 1, which does not set alignment
- Changed how HDF5 and PnetCDF calculate performance -- formerly each used
  the size of the stat()'ed file; changed it to be number of data bytes
  transferred.  these library-generated files can have large headers and
  filler as well as sparse file content
- Found potential overflow error in cast -- using IOR_offset_t, not int now
- Replaced HAVE_HDF5_NO_FILL macro to instead directly check if H5_VERS_MAJOR
  H5_VERS_MINOR are defined; if so, then must be HDF5-1.6.x or higher for
  no-fill usage.
- Corrected IOR_GetFileSize() function to point to HDF5 and NCMPI versions of
  IOR_GetFileSize() calls
- Changed the netcdf dataset from 1D array to 4D array, where the 4 dimensions
  are: [segmentCount][numTasks][numTransfers][transferSize]
  This patch from Wei-keng Liao allows for file sizes > 4GB (provided no
  single dimension is > 4GB).
- Finalized random-capability release
- Changed statvfs() to be for __sun (SunOS) only
- Retired Python GUI

Version 2.10.0.1
--------------------------------------------------------------------------------
- Cleaned up WriteOrRead(), reducing much to smaller functions.
- Added random capability for transfer offset.
- Modified strtok(NULL, " \t\r\n") in ExtractHints() so no trailing characters
- Added capability to set hints in NCMPI

Version 2.9.6.1
--------------------------------------------------------------------------------
- For 'pvfs2:' filename prefix, now skips DisplayFreeSpace(); formerly this
  caused a problem with statvfs()
- Changed gethostname() to MPI_Get_processor_name() so for certain cases
  when gethostname() would only return the frontend node name
- Added SunOS compiler settings for makefile
- Updated O_DIRECT usage for SunOS compliance
- Changed statfs() to instead use statvfs() for SunOS compliance
- Renamed compiler directive _USE_LUSTRE to _MANUALLY_SET_LUSTRE_STRIPING

Version 2.9.5
--------------------------------------------------------------------------------
- Wall clock deviation time relabeled to be "Start time skew across all tasks".
- Added notification for "Using reorderTasks '-C' (expecting block, not cyclic,
  task assignment)"
- Corrected bug with read performance with stonewalling (was using full size,
  stat'ed file instead of bytes transferred).

Version 2.9.4
--------------------------------------------------------------------------------
- Now using IOR_offset_t instead of int for tmpOffset in IOR.c:WriteOrRead().
  Formerly, this would cause error in file(s) > 2GB for ReadCheck.  The
  more-commonly-used WriteCheck option was not affected by this.

Version 2.9.3
--------------------------------------------------------------------------------
- Changed FILE_DELIMITER from ':' to '@'.
- Any time skew between nodes is automatically adjusted so that all nodes
  are calibrated to the root node's time.
- Wall clock deviation time is still reported, but have removed the warning
  message that was generated for this.  (Didn't add much new information,
  just annoyed folks.)
- The '-j' outlierThreshold option now is set to 0 (off) as default.  To use
  this, set to a positive integer N, and any task who's time (for open,
  access, etc.) is not within N seconds of the mean of all the tasks will show
  up as an outlier.

Version 2.9.2
--------------------------------------------------------------------------------
- Simple cleanup of error messages, format, etc.
- Changed error reporting so that with VERBOSITY=2 (-vv) any error encountered
  is displayed.  Previously, this was with VERBOSITY=3, along with full test
  parameters, environment, and all access times of operations.
- Added deadlineForStonewalling option (-D). This option is used for measuring
  the amount of data moved in a fixed time.  After the barrier, each task
  starts its own timer, begins moving data, and the stops moving data at a
  prearranged time.  Instead of measuring the amount of time to move a fixed
  amount of data, this option measures the amount of data moved in a fixed
  amount of time.  The objective is to prevent tasks slow to complete from
  skewing the performance.

Version 2.9.1
--------------------------------------------------------------------------------
- Updated test script to run through file1:file2 cases.
- Program exit code is now total numbers of errors (both writecheck and
  readcheck for all iterations), unless quitOnError (-q) is set.
- For various failure situations, replace abort with warning, including:
  - failed uname() for platform name now gives warning
  - failed unlink() of data file now gives warning
  - failed fsync() of data file now gives warning
  - failed open() of nonexistent script file now gives warning
- Changed labelling for error checking output to be (hopefully) more
  clear in details on errors for diagnostics.
- Another fix for -o file1:file2 option.
- Corrected bug in GetTestFileName() -- now possible to handle -o file1:file2
  cases for file-per-proc.

Version 2.9.0
--------------------------------------------------------------------------------
- Improved checkRead option to reread data from different node (to avoid cache)
  and then compare both reads.
- Added outlierThreshold (-j) option to give warning if any task is more than
  this number of seconds from the mean of all participating tasks.  If so, the
  task is identified, its time (start, elapsed create, elapsed transfer,
  elapsed close, or end) is reported, as is the mean and standard deviation for
  all tasks.  The default for this is 5, i.e. any task not within 5 seconds of
  the mean for those times is displayed.  This value can be set with
  outlierThreshold=<value> or -j <value>.
- Correct for clock skew between nodes - if skew greater than wallclock
  deviation threshold (WC_DEV_THRESHOLD) in seconds, then broadcast root
  node's timestamp and adjust by the difference.  WC_DEV_THRESHOLD currently
  set to 5.
- Added a Users Guide.