Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ConsoleLauncher to redirect STDOUT and STDERR to files instead of the console #3166

Open
tomwhoiscontrary opened this issue Feb 21, 2023 · 15 comments · May be fixed by #3637
Open

Allow ConsoleLauncher to redirect STDOUT and STDERR to files instead of the console #3166

tomwhoiscontrary opened this issue Feb 21, 2023 · 15 comments · May be fixed by #3637

Comments

@tomwhoiscontrary
Copy link

My test suites are large, and write a lot of stuff to output as they go along (mostly from the code under test rather than the tests themselves). I run them using ConsoleLauncher. The tree report at the end of the run is useful. But the torrent of console output is usually not: it is very helpful to have output captured when diagnosing failures, but seeing reams out output for passing tests is not.

I believe that JUnit can capture standard output and error. It would be very useful if there was a flag i could give to ConsoleLauncher to tell it to do this, and write the results to one or more files, instead of the console. The simplest thing would be to write all output to a single file (whose contents would be the same as what i currently see in the console). Significantly more useful would to write the output of each test to its own file, to make it easier to find. Perhaps tests which write no output could not create files at all, not sure.

A potential quirk here is that much of this output is written using a logging framework (usually java.util.logging) with an appender which writes to standard output. It would be possible to configure logging differently in tests, but i would prefer not to have to do that - just capturing output as it is written with the default logging configuration is greatly preferable to me.

One thing that wouldn't be useful to me is separate files for standard output and standard error. There is not usually a huge semantic difference between the two, whereas the relative order of messages written across both is usually very important, and separating the files would discard that.

Is there already some way to do this? Would it be possible for me to write some sort of extension to do this, which i could use without modifying my test sources? If not, would this make a good feature?

@marcphilipp
Copy link
Member

@tomwhoiscontrary Do you have a concrete proposal how this would be configured? Via new options on the ConsoleLauncher? Is there another CLI tool we could take inspiration from wrt. naming?

@tomwhoiscontrary
Copy link
Author

Well, my first thought is that if there is some way i could do this myself (a plug-in sort of thing), then i could play around with it and come back with a more concrete suggestion.

The simplest thing that would address my use case would be a flag to ConsoleLauncher like:

      --redirect-streams=FILE   Specify a path to a file to which standard output and error from tests will be redirected

I'm not very confident about the name. Sadly i can't think of any other tools which support this. Perhaps the methods in java.lang.ProcessBuilder are a precedent?

@marcphilipp
Copy link
Member

I think we should have separate options for stdout and stderr. Taking ProcessBuilder as an example, --redirect-output and --redirect-error would make sense. WDYT?

@tomwhoiscontrary
Copy link
Author

That would be fine for me - as long as i can redirect both to the same file!

@marcphilipp
Copy link
Member

Sure, I think we can make that work by checking if both are set to the same file.

@marcphilipp
Copy link
Member

@tomwhoiscontrary Would you be interested in taking a stab at this?

@tomwhoiscontrary
Copy link
Author

tomwhoiscontrary commented May 29, 2023

I am still interested in working on this! Just don't have a lot of time for it.

I started by doing a bit of reading and thinking.

Bits of the JDK where we should either capture output, or document that we can't capture output, etc:

  • System.out + System.err
  • java.io.Console
  • java.util.logging.ConsoleHandler
  • System.setOut + System.setErr
  • Threads used inside tests

Gradle captures output from tests, so it might be a good benchmark to try to do whatever Gradle does. I tried to find out how it does whatever it does, but the Gradle codebase is absolutely labyrinthine; i think the key point is DefaultStandardOutputRedirector, but i'm not sure.

Also interesting is the RedirectStdOutAndErr JUnit rule they use in testing.

I might get a chance to actually try some code next weekend ... Just remembered there's a film festival on!

@marcphilipp
Copy link
Member

Output capturing is already implemented in JUnit (as report entries): https://junit.org/junit5/docs/current/user-guide/#running-tests-capturing-output

I think we should try reusing that in some form but avoid the indirection of capturing it as strings.

@ShwetaliShimangaud
Copy link

I am interested in working on this problem. I am new to open-source contribution but has experience of working in java.
@marcphilipp and @tomwhoiscontrary, Can I look into this issue?

@tomwhoiscontrary
Copy link
Author

@ShwetaliShimangaud I have not worked on this at all, so please go ahead.

@ibrahimesseddyq
Copy link

ibrahimesseddyq commented Dec 12, 2023

ShwetaliShimangaud

collab? @ShwetaliShimangaud

@ibrahimesseddyq
Copy link

@marcphilipp should we care about thread-safety?

@mobounya
Copy link
Contributor

I've tried using output capturing as a ReportEntry, I'm getting the report in the method reportingEntryPublished in a TestExecutionListener, this listener is loaded from the ServiceLoader, the problem is I don't how can I pass data like the directory where I should put the file(s) in, or if we take the route of redirecting all of stdout to a single file like this --redirect-output=FILE I don't know how can I pass the name of the file from the command line to the execution listener, also I don't know how can I avoid capturing outputs as a String, any help would be highly appreciate it.

@mobounya
Copy link
Contributor

Update: we can deactivate a TestExecutionListener that's loaded from a ServiceLoader using a configuration parameter : https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom-deactivation

This can help us deactivate our TestExecutionListener that redirects output if no redirection CLI argument is found.

mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Add method registerMergedStandardStreams in StreamInterceptor to merge
stdout and stderr so both can be intercepted in an stdout interceptor.
This will keep the relative order for both outputs, which makes it
easier to correlate error messages with the corresponding output.

Add new configuration parameter junit.platform.output.capture.merge to
merge stdout and stderr and publish it as STDOUT_REPORT_ENTRY_KEY.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Add new CLI options --redirect-stdout and --redirect-stderr to
redirect stdout and stderr to a file, if both are set to the same file,
stdout and stderr will be merged so we can keep the relative order of
messages. This makes it easier to correlate error messages with the
corresponding output.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Document the changes for the previous two commit in the 5.11.0-M1
release notes and in the user guide.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Add method registerMergedStandardStreams in StreamInterceptor to merge
stdout and stderr so both can be intercepted in an stdout interceptor.
This will keep the relative order for both outputs, which makes it
easier to correlate error messages with the corresponding output.

Add new configuration parameter junit.platform.output.capture.merge to
merge stdout and stderr and publish it as STDOUT_REPORT_ENTRY_KEY to
all registered TestExecutionListener instances.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Add new CLI options --redirect-stdout and --redirect-stderr to
redirect stdout and stderr to a file, if both are set to the same file,
stdout and stderr will be merged so we can keep the relative order of
messages. This makes it easier to correlate error messages with the
corresponding output.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Document the changes for the previous two commit in the 5.11.0-M1
release notes and in the user guide.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 13, 2024
Document the changes for the previous two commit in the 5.11.0-M1
release notes and in the user guide.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 14, 2024
Add @API annotation to new methods.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 14, 2024
Add method registerMergedStandardStreams in StreamInterceptor to merge
stdout and stderr so both can be intercepted in an stdout interceptor.
This will keep the relative order for both outputs, which makes it
easier to correlate error messages with the corresponding output.

Add new configuration parameter junit.platform.output.capture.merge to
merge stdout and stderr and publish it as STDOUT_REPORT_ENTRY_KEY to
all registered TestExecutionListener instances.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 14, 2024
Add new CLI options --redirect-stdout and --redirect-stderr to
redirect stdout and stderr to a file, if both are set to the same file,
stdout and stderr will be merged so we can keep the relative order of
messages. This makes it easier to correlate error messages with the
corresponding output.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 14, 2024
Document the changes for the previous two commit in the 5.11.0-M1
release notes and in the user guide.

Issue: junit-team#3166
mobounya pushed a commit to mobounya/junit5 that referenced this issue Jan 14, 2024
Add @API annotation to new methods.

Issue: junit-team#3166
@sbrannen sbrannen changed the title Allow ConsoleLauncher to send standard output and error to files instead of the console Allow ConsoleLauncher to redirect STDOUT and STDERR to files instead of the console Jan 14, 2024
@sbrannen
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants