Generate PDF report for cypress + environment management improvements #1079

roicarrera · 2024-11-06T06:49:31Z

added mochawesome
added pdf-generator script file
read baseurl on config maps based on environment
added skip for prod

added functions for login with MFA added example and explained configuration

Login with MFA with Cypress

Improve cypress SUT environment management

added pdf generator script

generate pdf report

Backup branch

cschweikert · 2024-11-06T12:48:51Z

Hi @roicarrera, since this PR is also reworking parts of the respective configs, I would propose to use cypress.config.ts as base configuration that is then mixed into the other configs (import the base config and using the spread operator). This would prevent/remove a lot of code duplication and put more focus on the actual difference between those files.

e2e-cypress/files/cypress-acceptance.config.ts

e2e-cypress/files/cypress-integration.config.ts

e2e-cypress/files/package.json

e2e-cypress/files/pdf-generator.ts

e2e-cypress/files/support/test-evidence.ts

e2e-cypress/Jenkinsfile.template

e2e-cypress/files/support/e2e.ts

e2e-cypress/files/support/test-evidence.ts

reworked config files to be clearer fixed typo in reporterOptions added generate:pdf script to package.json moved folder creation

cschweikert

Thanks for making all the changes. Really improved things around the configs! Only little things left.

e2e-cypress/files/support/e2e.ts

e2e-cypress/files/pdf-generator.ts

e2e-cypress/files/support/test-evidence.ts

cschweikert

🚀

metmajer

Hello @roicarrera,

We need a clear concept for generating reports, screenshots, and videos from our testing Quickstarters. I cannot see clearly how their creation and storing does not affect Jenkins. The only place to store potentially larger files is Nexus.
Disabling test execution in PROD in the way you did seriously messes with regulations. A solid design involving the core platform team needs to be done to prevent potential damage.

As a general rule, I also suggest creating one PR for one purpose, and not mixing many different things. This will help increase clarity during code reviews.

metmajer · 2024-11-11T18:35:45Z

e2e-cypress/Jenkinsfile.template

+  // including potential downtime, data corruption, or exposure of sensitive information.
+  // This block is designed to skip tests in the production environment to avoid these risks.
+  // If you choose to enable testing in production, do so at your own risk and take all necessary precautions.
+  if (context.environment == 'prod') {


@roicarrera the Cypress Quickstarter is of type ods-test, which must support acceptance, integration, and installation tests. You cannot simply omit all tests in production. Since installation tests are mandatory by LeVA in the PROD environment, you are right now messing with compliance regulations. Changing the current testing behavior in PROD needs a clear design in agreemtn with the core platform team and consistency across all ods-test Quickstarters. FYI @BraisVQ

@roicarrera your comment says that testing in production means to take a risk and requires precaution. What do you mean? We made this conciously to ensure that applications, once in production, work as expected although tests are not strictly required. Limiting test execution to installation tests is a possible and helpful improvement for the future for sure.

metmajer · 2024-11-11T18:45:43Z

e2e-cypress/Jenkinsfile.template

@@ -49,27 +60,32 @@ odsComponentPipeline(

 def stageTest(def context) {
  stage('Integration Test') {
-    // OPTIONAL: load environment variables for Azure SSO with MSALv2; please adapt variable names to your OpenShift config
+    // Define your DEV and QA base URLs in a config map in OpenShift; please adapt variable names to your OpenShift config


@roicarrera why define the DEV and QA base urls in a secret if 12-factor apps teaches us to provide them through the environment? I believe that ODS already does that. Then, a test case should only reference a path and resouce below the BASE URL. Can you clarify the design? What about PROD?

metmajer · 2024-11-11T18:55:14Z

e2e-cypress/Jenkinsfile.template

+      sh 'npm run generate:pdf'
+      zip zipFile: 'cypress/pdf.zip', archive: false, dir: 'build/test-results/mochawesome/pdf'
+      archiveArtifacts artifacts: 'cypress/pdf.zip', fingerprint: true, daysToKeep: 2, numToKeep: 3
+
      if (fileExists('cypress/videos')) {
        zip zipFile: 'cypress/videos.zip', archive: false, dir: 'cypress/videos'
        stash(name: "acceptance-test-videos-${context.componentId}-${context.buildNumber}", includes: 'cypress/videos.zip', allowEmpty: true)
        archiveArtifacts artifacts: 'cypress/videos.zip', fingerprint: true, daysToKeep: 2, numToKeep: 3
      }

-      if (fileExists('cypress/screenshots')) {
-        zip zipFile: 'cypress/screenshots.zip', archive: false, dir: 'cypress/screenshots'
+      if (fileExists('build/test-results/screenshots')) {
+        zip zipFile: 'cypress/screenshots.zip', archive: false, dir: 'build/test-results/screenshots'
        stash(name: "acceptance-test-screenshots-${context.componentId}-${context.buildNumber}", includes: 'cypress/screenshots.zip', allowEmpty: true)
        archiveArtifacts artifacts: 'cypress/screenshots.zip', fingerprint: true, daysToKeep: 2, numToKeep: 3


@roicarrera what's the intended purpose of mochawesome reports opposed to LeVA-based documentation? Why archive reports, videos, and screenshots in Jenkins where they keep consuming disk space, even only for two days, which could cause Jenkins to fail when the disk runs full?

metmajer · 2024-11-12T10:14:43Z

e2e-cypress/Jenkinsfile.template

+    // cypressUser = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_username}' | base64 -d")
+    // cypressPassword = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_password}' | base64 -d")


@roicarrera credentials (secrets of type username and password) need to be consumed in a safe way so that they do not leak into the log stream. Please use Jenkins' withCredentials task. You can refer to other Quickstarters to see its use.

cschweikert · 2024-11-13T10:04:45Z

* We need a clear concept for generating reports, screenshots, and videos from our testing Quickstarters. I cannot see clearly how their creation and storing does not affect Jenkins. The only place to store potentially larger files is Nexus.

Build artifacts like screenshots and videos also fit the purpose of analyzing problems in case of broken tests. Yes, you should assure that e2e tests are successful BEFORE you commit. This could be done by running them on a locally running representation of your target system. However, the situation on the cluster is different. In rare cases you are even facing things like flaky tests that are hard to reproduce even on the cluster. In those situations as a developer you are thankful for every piece of information you could get.

In this context they fit where they are. However, we should discuss about the right amount of "daysToKeep". Also it would be great to understand, if storing screenshots and videos temporarily even creates a problem regarding storage sizes. Maybe Jenkins already takes care of this.

roicarrera and others added 13 commits October 18, 2024 12:39

removed obsolete msalv2-login_obsolete.ts

90ed448

added functions for login with MFA added example and explained configuration

updated documentation

ab435c4

Merge pull request #4 from roicarrera/feature/otp_login_for_cypress

a045724

Login with MFA with Cypress

updated CHANGELOG.md

0c2709f

added baseurl management based on environment

f41f1b2

add example of environment variables based on environment

4373a46

Merge branch 'master' into master

de05223

Merge pull request #5 from roicarrera/feature/cypress-env-improvements

cd122b6

Improve cypress SUT environment management

added mochawesome reporter + multi reporters

ab65f5a

added pdf generator script

applied dependabot changes from ods

c5ca704

Merge pull request #6 from roicarrera/feature/generate-pdf-report

53d2bc8

generate pdf report

Merge branch 'master' into backup-branch

ae8e16f

Merge pull request #7 from roicarrera/backup-branch

3dca248

Backup branch

roicarrera requested a review from cschweikert as a code owner November 6, 2024 06:49

roicarrera added 2 commits November 6, 2024 07:50

update CHANGELOG.md

0fc0fb7

Merge branch 'master' of https://github.com/roicarrera/ods-quickstarters

920717b

cschweikert requested changes Nov 6, 2024

View reviewed changes

cschweikert reviewed Nov 6, 2024

View reviewed changes

e2e-cypress/files/support/e2e.ts Outdated Show resolved Hide resolved

e2e-cypress/files/support/test-evidence.ts Show resolved Hide resolved

adressed comments in PR

cf20766

reworked config files to be clearer fixed typo in reporterOptions added generate:pdf script to package.json moved folder creation

cschweikert reviewed Nov 11, 2024

View reviewed changes

e2e-cypress/files/support/e2e.ts Outdated Show resolved Hide resolved

e2e-cypress/files/pdf-generator.ts Outdated Show resolved Hide resolved

e2e-cypress/files/support/test-evidence.ts Outdated Show resolved Hide resolved

small fixes

9f1bd9e

cschweikert approved these changes Nov 11, 2024

View reviewed changes

metmajer requested changes Nov 11, 2024

View reviewed changes

metmajer requested changes Nov 12, 2024

View reviewed changes

This was referenced Nov 12, 2024

Cypress evidence improvements #1065

Merged

Cypress login with MFA #1070

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate PDF report for cypress + environment management improvements #1079

Generate PDF report for cypress + environment management improvements #1079

roicarrera commented Nov 6, 2024

cschweikert commented Nov 6, 2024

cschweikert left a comment

cschweikert left a comment

metmajer left a comment •

edited

Loading

metmajer Nov 11, 2024

metmajer Nov 12, 2024 •

edited

Loading

metmajer Nov 11, 2024

metmajer Nov 11, 2024

metmajer Nov 12, 2024

cschweikert commented Nov 13, 2024

		// cypressUser = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_username}' \| base64 -d")
		// cypressPassword = sh(returnStdout: true, script:"oc get secret e2euser -o jsonpath='{.data.${context.environment}_password}' \| base64 -d")

Generate PDF report for cypress + environment management improvements #1079

Are you sure you want to change the base?

Generate PDF report for cypress + environment management improvements #1079

Conversation

roicarrera commented Nov 6, 2024

cschweikert commented Nov 6, 2024

cschweikert left a comment

Choose a reason for hiding this comment

cschweikert left a comment

Choose a reason for hiding this comment

metmajer left a comment • edited Loading

Choose a reason for hiding this comment

metmajer Nov 11, 2024

Choose a reason for hiding this comment

metmajer Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

metmajer Nov 11, 2024

Choose a reason for hiding this comment

metmajer Nov 11, 2024

Choose a reason for hiding this comment

metmajer Nov 12, 2024

Choose a reason for hiding this comment

cschweikert commented Nov 13, 2024

metmajer left a comment •

edited

Loading

metmajer Nov 12, 2024 •

edited

Loading