Spike API for attaching files to test executions #3336

marcphilipp · 2023-05-30T19:20:10Z

No description provided.

marcphilipp · 2023-05-30T19:21:13Z

documentation/src/test/java/example/TestReporterDemo.java

+	@Test
+	void reportFiles(TestReporter testReporter, @TempDir Path tempDir) throws Exception {
+
+		testReporter.publishFile("test1.txt", file -> Files.write(file, singletonList("Test 1")));
+
+		Path existingFile = tempDir.resolve("test2.txt");
+		testReporter.publishFile(Files.write(existingFile, singletonList("Test 2")));
+
+		testReporter.publishFile("test3", dir -> {
+			Path nestedFile = Files.createDirectory(dir).resolve("nested.txt");
+			Files.write(nestedFile, singletonList("Nested content"));
+		});
+	}


This demonstrates what the programming model for test written with JUnit Jupiter would look like.

marcphilipp · 2023-05-30T19:24:32Z

junit-jupiter-api/src/main/java/org/junit/jupiter/api/TestReporter.java

+	 */
+	@API(status = EXPERIMENTAL, since = "5.11")
+	default void publishFile(String fileName, ThrowingConsumer<Path> action) {
+		throw new UnsupportedOperationException();


This interface shouldn't be implemented by clients but is annotated with @FunctionalInterface so adding another abstract method would, at least theoretically, break backward compatibility. We could decide to ignore that in this case since nobody should have implemented this, go with this pattern, or introduce a separate FileReporter interface that can be injected alongside TestReporter or could extend it and replace it.

marcphilipp · 2023-05-30T19:26:09Z

...ng/src/main/java/org/junit/platform/reporting/open/xml/OpenTestReportGeneratingListener.java

+		eventsFileWriter.append(reported(id, Instant.now()), //
+			reported -> reported.append(attachments(), attachments -> attachments.append(file(), file -> {
+				file.withTime(entry.getTimestamp());
+				file.withPath(outputDir.toPath().relativize(entry.getFile()).toString());
+			})));


This is reported like this in the XML file:

<e:reported id="25" time="2023-05-30T19:00:31.623824Z"> <attachments> <junit:file path="junit-5012861544611183477/junit-jupiter/example.TestReporterDemo/reportFile(org.junit.jupiter.api.TestReporter, java.nio.file.Path)/test1.txt" time="2023-05-30T21:00:31.621204"/> </attachments> </e:reported>

Instead of the junit namespace, this should be declared in the core namespace of open-test-reporting. I only did it here to avoid having to work in two repos for this spike.

marcphilipp · 2023-05-30T19:27:23Z

junit-platform-testkit/src/main/java/org/junit/platform/testkit/engine/EngineTestKit.java

-		ExecutionRequest request = new ExecutionRequest(engineTestDescriptor, listener,
-			discoveryRequest.getConfigurationParameters());
+		ExecutionRequest request = ExecutionRequest.create(engineTestDescriptor, listener,
+			discoveryRequest.getConfigurationParameters(), OutputDirProvider.NOOP);


For testing with EngineTestKit a special implementation of OutputDirProvider should be provided that uses a temporary directory that is cleaned up after the surrounding test is finished.

dsaff

This definitely feels like it's a good direction! My primary design question is whether we'd want a way to attach "types" (MIME types, or something else?) to these files.

In a separate thread of thought, how could we make this available for people still using JUnit 4?

junit-jupiter-api/src/main/java/org/junit/jupiter/api/TestReporter.java

...r-engine/src/main/java/org/junit/jupiter/engine/execution/JupiterEngineExecutionContext.java

dsaff · 2023-07-13T19:20:58Z

junit-platform-engine/src/main/java/org/junit/platform/engine/reporting/FileEntry.java

+ * @see #from(Path)
+ */
+@API(status = EXPERIMENTAL, since = "1.11")
+public final class FileEntry {


Debating whether we'd want some kind of MIME type, or if we'd trust report users to use extensions or content analysis to know which files were, for example, displayable as images...

I can see a case for an optional mime type but making it mandatory is probably too heavy a burden on users. I'd expect one would browse the output dir in a file explorer or web browser at some point which will rely mostly on file extensions anyway. We could add a mime type later or right away. WDYT?

Leaving it for later seems fine.

...m-launcher/src/main/java/org/junit/platform/launcher/core/HierarchicalOutputDirProvider.java

marcphilipp · 2023-07-29T10:06:32Z

This definitely feels like it's a good direction! My primary design question is whether we'd want a way to attach "types" (MIME types, or something else?) to these files.

see #3336 (comment)

In a separate thread of thought, how could we make this available for people still using JUnit 4?

I was wondering that as well. I'm hesitant to change JUnit 4's RunListener API to add a method that can publish a file. However, we could provide a TestRule implementation that has an API similar to the methods added in this PR to TestReporter. The rule would only work when used via the Vintage engine. Otherwise publishing files would be a no-op. The Vintage engine could provide a backing reporter mechanism via a ThreadLocal. If you think that might be viable, I can take a stab at spiking it.

marcphilipp · 2023-07-29T10:07:04Z

@leonard84 How would you expose this to Spock tests?

marcphilipp · 2023-07-29T13:16:15Z

The Vintage engine could provide a backing reporter mechanism via a ThreadLocal.

On second thought, maybe a ServiceLoader-based approach would be better.

marcphilipp · 2023-07-29T15:30:44Z

@dsaff I pushed a POC for how it could work for JUnit 4

marcphilipp · 2023-07-29T15:31:02Z

documentation/src/test/java/example/vintage/VintageTestReportingDemo.java

+public class VintageTestReportingDemo {
+
+	@Rule
+	public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+	@Rule
+	public TestReporting testReporting = new TestReporting();
+
+	@Test
+	public void reportFiles() throws Exception {
+		Path existingFile = Files.write(temporaryFolder.getRoot().toPath().resolve("test.txt"), singletonList("Test"));
+		testReporting.publishFile(existingFile);
+	}
+}


JUnit 4 example

Looks really clean!

leonard84 · 2024-03-15T12:57:54Z

junit-jupiter-api/src/main/java/org/junit/jupiter/api/TestReporter.java

+	 */
+	@API(status = EXPERIMENTAL, since = "5.11")
+	default void publishFile(Path file) {
+		publishFile(file.getFileName().toString(), path -> Files.copy(file, path, REPLACE_EXISTING));


I think it would make sense to move the file by default or at least provide an option to do so.

leonard84 · 2024-03-15T12:59:49Z

junit-jupiter-api/src/main/java/org/junit/jupiter/api/TestReporter.java

@@ -77,4 +82,36 @@ default void publishEntry(String value) {
 		this.publishEntry("value", value);
 	}

+	/**


Having a way to "add" a remote file, e.g., via a URI, would be useful.

leonard84 · 2024-03-15T13:05:15Z

junit-platform-engine/src/main/java/org/junit/platform/engine/reporting/FileEntry.java

+	}
+
+	private final LocalDateTime timestamp = LocalDateTime.now();
+	private final Path file;


Adding a Map<String, String> for metadata/annotations would be useful.
For example, you could store the URL for the screenshot's current page and other values.

marcphilipp added this to the 5.11 M1 milestone May 30, 2023

marcphilipp commented May 30, 2023

View reviewed changes

marcphilipp requested review from dsaff and leonard84 May 30, 2023 19:27

dsaff reviewed Jul 13, 2023

View reviewed changes

Spike API for attaching files to test executions

66c5dc0

marcphilipp force-pushed the marc/2296-file-attachments branch from f18a70e to 66c5dc0 Compare July 29, 2023 09:14

marcphilipp added 4 commits July 29, 2023 11:25

Clarify Javadoc

6256b39

Move OutputDirProvider to EngineDiscoveryRequest

170bfa6

Make example clearer

40e585b

Fix replacing of forbidden characters

fc073e3

marcphilipp requested a review from dsaff July 29, 2023 10:06

Add POC for JUnit 4 tests with Vintage engine

64c22de

marcphilipp commented Jul 29, 2023

View reviewed changes

marcphilipp modified the milestones: 5.11 M1, 5.12 Jan 12, 2024

marcphilipp modified the milestones: 5.12, 5.12 tmp Feb 2, 2024

marcphilipp self-assigned this Mar 9, 2024

leonard84 reviewed Mar 15, 2024

View reviewed changes

marcphilipp mentioned this pull request Mar 15, 2024

Add a schema for attachments ota4j-team/open-test-reporting#3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spike API for attaching files to test executions #3336

Spike API for attaching files to test executions #3336

marcphilipp commented May 30, 2023

marcphilipp May 30, 2023

marcphilipp May 30, 2023

marcphilipp May 30, 2023

marcphilipp May 30, 2023

dsaff left a comment

dsaff Jul 13, 2023

marcphilipp Jul 29, 2023

dsaff Mar 13, 2024

marcphilipp commented Jul 29, 2023

marcphilipp commented Jul 29, 2023

marcphilipp commented Jul 29, 2023

marcphilipp commented Jul 29, 2023

marcphilipp Jul 29, 2023

dsaff Mar 13, 2024

leonard84 Mar 15, 2024

leonard84 Mar 15, 2024

leonard84 Mar 15, 2024

Spike API for attaching files to test executions #3336

Are you sure you want to change the base?

Spike API for attaching files to test executions #3336

Conversation

marcphilipp commented May 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsaff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcphilipp commented Jul 29, 2023

marcphilipp commented Jul 29, 2023

marcphilipp commented Jul 29, 2023

marcphilipp commented Jul 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment