add command line flag for system test result dumps

[efd6]

Please take a look.

[mrodm]

Adding just one flag it forces to always set a prefix to the user, but probably they often need just a default value for that.

What about adding two flags to manage this feature?

• One flag to enable or not dumping the documents (--dump-docs) without defining any path or prefix.
  • The default filename could be the same one used when the environment variable is set.
  • Should this file be saved in the same location where elastic-package runs ? Take into account that this could change with the parameter -C.

    elastic-package -C test/packages/parallel/nginx test system -v

• Another flag to set a custom prefix (--dump-docs-prefix) or even it could be set the whole path (--dump-docs-output) leaving to the user the choice to where save the output file. I think I'd prefer the latter.

# enable dumping documents and using the default filename for the path
elastic-package test system -v --dump-docs-

# enable dumping documents and using a custom path
elastic-package test system -v --dump-docs --dump-docs-output /path/to/dump.json

Doing it like this, it would be similar on how coverage is set in elastic-package:

• --test-coverage to enable the feature,
• --coverage-format to choose which format is required.

[efd6]

I think this comes down to a philosophical position; I think having a default location for things like this is a significant misfeature. IMO actions should require explicit requests. Including things like this in the config add complexity to the user experience and is also IMO a misfeature.

I'm not entirely sure what the gain is for users in adding this additional complexity.

[mrodm]

IMO I think most users (from my POV) would like just to download/dump the documents without really requiring setting any specific path or prefix. That's why I would say that the most common usage would be (not requiring to set any prefix there):

elastic-package test system -v --dump-docs

Moreover, doing it like this, it would enable this feature and it could be coded to be equivalent to set the ELASTIC_PACKAGE_TEST_DUMP_SCENARIO_DOCS environment variable. So users setting the environment variable and setting that flag can expect the same exact behavior of elastic-package.

For the other flag I proposed, I thought there could be users that require some specific known file, for instance they are running tests with different packages and they want those files to be easily identified. That's my reasoning to add that new --dump-docs-output or --dump-docs-prefix flag, so they can decide how to save those documents.

# allow a full path
elastic-package test system -v --dump-docs --dump-docs-output /path/store/documents.json

# or as you proposed adding a prefix and save them in the same location where elastic-package runs
elastic-package test system -v --dump-docs --dump-docs-prefix "package-test"

The concern I have of writing those new documents in the same location as elastic-package runs is that this could lead to users committing and pushing those files in the repository inadvertently.

[efd6]

IMO I think most users (from my POV) would like just to download/dump the documents without really requiring setting any specific path or prefix.

This is certainly not what I'd want; doing this means that I cannot script obtaining the data.

In terms of work, specifying a location is easier than digging through logs to find the name that the program chose. In my ideal world, the output would not be to a file whose name was under the control of the program at all. IMO this should be entirely under the control of the user, but I made a concession to the current behaviour on the basis that it was still relatively easy to obtain the files if the prefix was known.

[mrodm]

Ok, got it.

Could it be changed what the value would indicate for this flag? Instead of setting a prefix for files, this flag could set the path to a folder.

I'm trying to be consistent with other commands/parameters used in elastic-package. For instance, in these commands --output is a path to a folder:

• dump stack logs: elastic-package stack dump --output <path>
• dump assets: elastic-package dump agent-policies --output <path>

That folder should be checked (and created if needed) by elastic-package. And it would be the folder where the JSON files are stored.
Files stored in that folder could follow the same naming patterns as it was set when setting the environment variable:

    dumpPath = filepath.Join(dumpFolderBase, fmt.Sprintf("elastic-package-test-docs-dump-%s.json", time.Now().Format("20060102150405")))

So, elastic-package would be like this:

elastic-package test system -v --dump-logs "docs_folder_package/"

Probably, doing it like this, the parameter could be renamed as --dump-logs-to.

[efd6]

I'm trying to be consistent with other commands/parameters used in elastic-package.

ISTM that to be consistent with those the flag should be called dump.

That folder should be checked (and created if needed) by elastic-package.

I think in many areas ep does too much (I'm a strong follower of the unix philosophy), creating paths is a thing that is done by mkdir.

elastic-package test system -v --dump-logs "docs_folder_package/"

I think that I'd be OK with stating the path and determining whether it's a directory and using this behaviour in that case, otherwise using the behaviour that exists in this change. Note that the dumped artefacts are not logs in the general case, so naming the flag that would be misleading/confusing (I'm already thinking that "dump" is a very long spelling of "o").

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: eb7be7f

History

• 💔 Build #6274 failed 9456b6f
• 💚 Build #6266 succeeded a9fe9ba
• 💔 Build #6265 failed cd7a5e3

cc @efd6