Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add comments to PRs when builds/tests fail and Develocity build scans are enabled #271

Open
yrodiere opened this issue Sep 12, 2024 · 0 comments

Comments

@yrodiere
Copy link
Member

When a test fails, it can be helpful to add a comment to the PR to specify exactly what failed, with appropriate links.

When using the Develocity integration, we could possibly do it:

  1. On the first failed build, we create a comment on the PR (see below for content) -- and include a hidden HTML marker (HTML comment) containing the commit hash this PR comment refers to.
  2. On subsequent failed builds, we look for comments from the bot on the PR, and minimize those whose HTML marker refers a different (older) commit hash. If we find a PR comment containing the HTML marker for the current commit, we update that PR comment; if we don't find one, we create a new PR comment.

The content of the PR comment could take inspiration from what Guillaume did on Quarkus. But:

  1. We'd want a single comment for all "workflows/jobs" (GitHub Actions/Jenkins)
  2. The code from Quarkus is based on Maven Surefire output and cannot (easily) be reused for Gradle-based builds.
  3. So I'd suggest to base all our data on Develocity metadata: I think the Develocity APIs allow us to list failing tests, so that would be a great start?

Something like the example below would be a start. Note the idea with the tests is to present, for each failed test, links to test failures as well as test successes, labelling both failures and tests with relevant Develocity Build Scan labels.


3 builds failed

Job/Workflow OS Java DB Goals/Tasks Results
Hibernate ORM build Linux 17 sybase check 🔴 📃
Hibernate ORM build Linux 17 mssql check 🔴 📃
Hibernate ORM build Linux 17 mysql check 🔴 📃

12 builds succeeded

Job/Workflow OS Java DB Goals/Tasks Results
Hibernate ORM build Linux 17 hsqldb check 🟢 📃
Hibernate ORM build Linux 17 h2 check 🟢 📃
Hibernate ORM build Linux 17 edb check 🟢 📃
Hibernate ORM build Linux 17 pgsql check 🟢 📃
Hibernate ORM build Linux 17 mariadb check 🟢 📃
Hibernate ORM build Linux 17 oracle check 🟢 📃
Hibernate ORM build Linux 17 db2 check 🟢 📃
Hibernate ORM build-Atlas Linux 21 oracle_cloud_autonomous check 🟢 📃
Hibernate ORM build-Atlas Linux 21 oracle_cloud_db19c check 🟢 📃
Hibernate ORM build-Atlas Linux 21 oracle_cloud_db23c check 🟢 📃
Hibernate ORM build-Atlas Linux 21 oracle_cloud_db21c check 🟢 📃
hibernate-orm-pipeline/PR-8932 Linux 17 tidb check 🟢 📃

14 tests failed

:hibernate-core:test

Test Failures ❌ Successes ✅

org.hibernate.orm.test.mapping.onetomany.OneToManyOrderedListJoinTableTest

jdk-17 sybase
jdk-17 mssql
jdk-17 mysql

jdk-17 hsqldb
jdk-17 h2
jdk-21 h2
jdk-23 h2
jdk-17 edb

testAddAtZeroDeleteAtTwo(SessionFactoryScope)

jdk-17 sybase
jdk-17 mysql

jdk-17 mssql
jdk-17 hsqldb
jdk-17 h2
jdk-21 h2
jdk-23 h2
jdk-17 edb

testNormalAddAndRemove(SessionFactoryScope)

jdk-17 mssql
jdk-17 mysql

jdk-17 sybase
jdk-17 hsqldb
jdk-17 h2
jdk-21 h2
jdk-23 h2
jdk-17 edb

testAddSameElementTwice(SessionFactoryScope)

jdk-17 sybase
jdk-17 mssql
jdk-17 mysql

jdk-17 hsqldb
jdk-17 h2
jdk-21 h2
jdk-23 h2
jdk-17 edb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant