Skip to content

Commit

Permalink
chore: Updated error handling (#36288)
Browse files Browse the repository at this point in the history 
## Description
Added extra checks for better reporting


Fixes #`Issue Number`  
_or_  
Fixes `Issue URL`
> [!WARNING]  
> _If no issue exists, please create an issue first, and check with the
maintainers if the issue is valid._

## Automation

/ok-to-test tags="@tag.ImportExport"

### 🔍 Cypress test results
<!-- This is an auto-generated comment: Cypress test results  -->
> [!TIP]
> 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
> Workflow run:
<https://github.com/appsmithorg/appsmith/actions/runs/10831607193>
> Commit: 2a5049d
> <a
href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=10831607193&attempt=1"
target="_blank">Cypress dashboard</a>.
> Tags: `@tag.ImportExport`
> Spec:
> <hr>Thu, 12 Sep 2024 13:31:10 UTC
<!-- end of auto-generated comment: Cypress test results  -->


## Communication
Should the DevRel and Marketing teams inform users about this change?
- [ ] Yes
- [x] No


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Enhanced CI workflow to provide clearer test results through improved
commenting on pull requests based on test outcomes.
- Introduced a failure flag mechanism to explicitly record test
execution status.

- **Bug Fixes**
- Improved reliability of CI process by ensuring important logs and
results are preserved and uploaded even on test failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • Loading branch information
@sagar-qa007
sagar-qa007 authored Sep 12, 2024
1 parent 7ba06c8 commit 71b44a1
Show file tree
Hide file tree
Showing 2 changed files with 93 additions and 34 deletions.
86 changes: 66 additions & 20 deletions .github/workflows/build-client-server-count.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ jobs:
with:
pr: ${{fromJson(needs.file-check.outputs.pr)}}
run_count: ${{fromJson(needs.file-check.outputs.run_count)}}


ci-test-limited-existing-docker-image:
needs: [file-check]
Expand Down Expand Up @@ -260,39 +261,62 @@ jobs:
with:
name: cypress-repeat-logs
path: app/client

- name: Download the ci_test_status
uses: actions/download-artifact@v4
with:
name: ci_test_status
path: app/client

- name: Read and Set File Content as ENV Variable
id: set-summary-content
run: |
summary_content=$(cat app/client/cy-repeat-summary.txt | tr '\n' ' ')
echo "summary_content=$summary_content" >> $GITHUB_ENV
echo "summary_content=$summary_content" >> $GITHUB_ENV
- name: Check CI Test Result
id: check-ci-test
run: |
# Check if the ci_test_failed flag is set
if [ -f app/client/ci_test_status.txt ]; then
if grep -q "ci_test_failed=true" app/client/ci_test_status.txt; then
echo "Tests failed. Please review the test results."
echo "ci_test_failed=true" >> $GITHUB_ENV
else
echo "Tests passed."
echo "ci_test_failed=false" >> $GITHUB_ENV
fi
else
echo "ci_test_status.txt file not found."
echo "ci_test_failed=unknown" >> $GITHUB_ENV
fi
- name: Add a comment on the PR with new CI failures
if: needs.ci-test-limited.result != 'success' && needs.file-check.outputs.pr != '0'
if: env.ci_test_failed == 'true' && needs.file-check.outputs.pr != '0'
uses: peter-evans/create-or-update-comment@v3
with:
issue-number: ${{fromJson(needs.file-check.outputs.pr)}}
body: |
Workflow run: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}>.
Cypress dashboard: <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=${{ github.run_id }}&attempt=${{ github.run_attempt }}&selectiontype=test&testsstatus=failed&specsstatus=fail" target="_blank"> Click here!</a>
The following are new failures, please fix them before merging the PR: ${{env.new_failed_spec_env}}
To know the list of identified flaky tests - <a href="https://internal.appsmith.com/app/cypress-dashboard/identified-flaky-tests-65890b3c81d7400d08fa9ee3?branch=master" target="_blank">Refer here</a>
```
${{ env.summary_content }}
```
- name: Add a comment on the PR when ci-test-limited is success
if: needs.ci-test-limited.result == 'success' && needs.file-check.outputs.pr != '0'
Workflow run: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}>.
Cypress dashboard: <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=${{ github.run_id }}&attempt=${{ github.run_attempt }}&selectiontype=test&testsstatus=failed&specsstatus=fail" target="_blank">Click here!</a>
The following are new failures, please fix them before merging the PR: ${{env.new_failed_spec_env}}
To know the list of identified flaky tests - <a href="https://internal.appsmith.com/app/cypress-dashboard/identified-flaky-tests-65890b3c81d7400d08fa9ee3?branch=master" target="_blank">Refer here</a>
```
${{ env.summary_content }}
```
- name: Add a comment on the PR when ci-test-limited is successful
if: env.ci_test_failed != 'true' && needs.file-check.outputs.pr != '0'
uses: peter-evans/create-or-update-comment@v3
with:
issue-number: ${{fromJson(needs.file-check.outputs.pr)}}
body: |
Workflow run: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}>.
Cypress dashboard url: <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=${{ github.run_id }}&attempt=${{ github.run_attempt }}" target="_blank">Click here!</a>
All cypress tests have passed 🎉🎉🎉
```
${{ env.summary_content }}
```
Workflow run: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}>.
Cypress dashboard url: <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=${{ github.run_id }}&attempt=${{ github.run_attempt }}" target="_blank">Click here!</a>
All Cypress tests have passed 🎉🎉🎉
```
${{ env.summary_content }}
```
- name: Check ci-test-limited set status
if: needs.ci-test-limited.result != 'success'
Expand Down Expand Up @@ -396,16 +420,38 @@ jobs:
with:
name: cypress-repeat-logs
path: app/client

- name: Download the ci_test_status
uses: actions/download-artifact@v4
with:
name: ci_test_status
path: app/client

- name: Read and Set File Content as ENV Variable
id: set-summary-content
run: |
summary_content=$(cat app/client/cy-repeat-summary.txt | tr '\n' ' ')
echo "summary_content=$summary_content" >> $GITHUB_ENV
- name: Check CI Test Result
id: check-ci-test
run: |
# Check if the ci_test_failed flag is set
if [ -f app/client/ci_test_status.txt ]; then
if grep -q "ci_test_failed=true" app/client/ci_test_status.txt; then
echo "Tests failed. Please review the test results."
echo "ci_test_failed=true" >> $GITHUB_ENV
else
echo "Tests passed."
echo "ci_test_failed=false" >> $GITHUB_ENV
fi
else
echo "ci_test_status.txt file not found."
echo "ci_test_failed=unknown" >> $GITHUB_ENV
fi
- name: Add a comment on the PR with new CI failures
if: needs.ci-test-limited-existing-docker-image.result != 'success' && needs.file-check.outputs.pr != '0'
if: env.ci_test_failed == 'true' && needs.file-check.outputs.pr != '0'
uses: peter-evans/create-or-update-comment@v3
with:
issue-number: ${{fromJson(needs.file-check.outputs.pr)}}
Expand All @@ -419,7 +465,7 @@ jobs:
```
- name: Add a comment on the PR when ci-test-limited-existing-docker-image is success
if: needs.ci-test-limited-existing-docker-image.result == 'success' && needs.file-check.outputs.pr != '0'
if: env.ci_test_failed != 'true' && needs.file-check.outputs.pr != '0'
uses: peter-evans/create-or-update-comment@v3
with:
issue-number: ${{fromJson(needs.file-check.outputs.pr)}}
Expand Down
41 changes: 27 additions & 14 deletions .github/workflows/ci-test-limited-with-count.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -350,14 +350,19 @@ jobs:
--spec ${{ env.specs_to_run }} \
--config-file "cypress_ci_custom.config.ts"
cat cy-repeat-summary.txt
# Check if "Total Failed: 0" is present
# Define the path for the failure flag file
FAILURE_FLAG_FILE="ci_test_status.txt"
# Check for test results and store the status in the file
if ! grep -q "Total Failed: 0" cy-repeat-summary.txt; then
echo "Tests failed, failing the GitHub Action."
exit 1 # Fails the step if tests failed
fi
echo "ci_test_failed=true" > "$FAILURE_FLAG_FILE"
else
echo "ci_test_failed=false" > "$FAILURE_FLAG_FILE"
fi
cat "$FAILURE_FLAG_FILE"
- name: Trim number of cypress log files
if: failure()
if: always()
run: |
find ${{ github.workspace }}/app/client/cypress/cypress-logs -name '*.json' -type f | tail -n +11 | xargs -I {} rm -- {}
Expand All @@ -368,9 +373,17 @@ jobs:
name: cypress-repeat-logs
path: ${{ github.workspace }}/app/client/cy-repeat-summary.txt
overwrite: true

- name: Upload ci_test_status.txt artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: ci_test_status
path: ${{ github.workspace }}/app/client/ci_test_status.txt
overwrite: true

- name: Upload failed test cypress logs artifact
if: failure()
if: always()
uses: actions/upload-artifact@v4
with:
name: cypress-console-logs
Expand All @@ -386,29 +399,29 @@ jobs:
overwrite: true

- name: Collect CI container logs
if: failure()
if: always()
working-directory: "."
run: |
mkdir -p ~/dockerlogs
docker logs appsmith 2>&1 > ~/dockerlogs/dockerlogs-log.txt
# Upload docker logs
- name: Upload failed test list artifact
if: failure()
if: always()
uses: actions/upload-artifact@v4
with:
name: dockerlogs
path: ~/dockerlogs
overwrite: true

- name: Rename reports
if: failure()
if: always()
run: |
mkdir -p ~/results
mv ${{ github.workspace }}/app/client/results ~/results
- name: Upload cypress report
if: failure()
if: always()
uses: actions/upload-artifact@v4
with:
name: results-${{github.run_attempt}}
Expand All @@ -417,13 +430,13 @@ jobs:

# Set status = failedtest
- name: Set fail if there are test failures
if: failure()
if: always()
run: |
echo "failedtest" > ~/run_result
# Force store previous run result to cache
- name: Store the previous run result
if: failure()
if: always()
uses: actions/cache/save@v4
with:
path: |
Expand All @@ -433,7 +446,7 @@ jobs:
# Upload the log artifact so that it can be used by the test & deploy job in the workflow
- name: Upload server logs bundle on failure
uses: actions/upload-artifact@v4
if: failure()
if: always()
with:
name: server-logs
path: app/server/server-logs.log
Expand Down

0 comments on commit 71b44a1

Please sign in to comment.