Skip to content

GH-49930: [CI][C++] Pin MinGW MSYS2 packages to unblock CI#49931

Open
tadeja wants to merge 5 commits intoapache:mainfrom
tadeja:ci-mingw-fix
Open

GH-49930: [CI][C++] Pin MinGW MSYS2 packages to unblock CI#49931
tadeja wants to merge 5 commits intoapache:mainfrom
tadeja:ci-mingw-fix

Conversation

@tadeja
Copy link
Copy Markdown
Contributor

@tadeja tadeja commented May 5, 2026

Rationale for this change

! Temporary workaround for #49930 !!
Both MinGW jobs fail every run since 2026 April 30, likely because of two MSYS2 updates:

  1. MINGW64: gcc 15.2 -> 16.1 made the intermittent __emutls race in ThreadPool always reproduce (see [C++][CI] arrow-json-test segfaults occasionally on AMD64 Windows MinGW MINGW64 C++ job #49272, fix in GH-49272: [C++][CI] Fix intermittent segfault in arrow-json-test with MinGW #49462).
  2. Both MINGW64 and CLANG64: aws-sdk-cpp 1.11.479 -> 1.11.801 stopped sending Content-Md5 on DeleteObjects; bundled MinIO RELEASE.2024-09-13 still requires it.

What changes are included in this PR?

CI change, new Pin MSYS2 packages step in cpp.yml

Are these changes tested?

Tested on a fork:
failing ->
passing

Are there any user-facing changes?

No

@github-actions github-actions Bot added the awaiting review Awaiting review label May 5, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

⚠️ GitHub issue #49930 has been automatically assigned in GitHub to PR creator.

@tadeja
Copy link
Copy Markdown
Contributor Author

tadeja commented May 6, 2026

With temporary CI changes here MinGW failing tests now pass...
Except for the single test arrow-flight-test Timeout on CLANG64 - could we rerun this? (@kou ?) https://github.com/apache/arrow/actions/runs/22742119641/job/65957707054?pr=49402#step:14:1080

arrow-flight-test has this issue already logged (for odbc) #49465

@tadeja
Copy link
Copy Markdown
Contributor Author

tadeja commented May 6, 2026

@raulcd do you agree we could temporarily use solution from this draft PR (= listed failures in newer, broader issue #49930) in order to resolve MinGW C++ failures on CI while waiting on stuck PR #49462 (= your older issue #49272)?

Copy link
Copy Markdown
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we faced this in the past on other jobs and we updated the minio version which contains the fix for newer aws-sdk-cpp. Shouldn't we just update minio version this is what we use on other jobs:

# Use specific versions for minio server and client to avoid CI failures on new releases.
minio_version="minio.RELEASE.2025-01-20T14-49-07Z"
mc_version="mc.RELEASE.2024-09-16T17-43-14Z"

As per the issue with the segfault, I think we should deal with it independently and possibly apply the proposed fix?

@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels May 6, 2026
@kou
Copy link
Copy Markdown
Member

kou commented May 6, 2026

It seems that some MinIO alternatives such as https://github.com/rustfs/rustfs .
Can we migrate from MinIO to solve this problem?

@tadeja
Copy link
Copy Markdown
Contributor Author

tadeja commented May 6, 2026

It seems that some MinIO alternatives such as https://github.com/rustfs/rustfs . Can we migrate from MinIO to solve this problem?

interesting point! @kou rok mentions there's this existing issue, perhaps best to discuss alternative there? #47908

@raulcd
Copy link
Copy Markdown
Member

raulcd commented May 6, 2026

It seems that some MinIO alternatives such as https://github.com/rustfs/rustfs . Can we migrate from MinIO to solve this problem?

For this specific issue I think that's unnecessary. The problem is that MinGW was using an old version of aws-sdk which was compatible with the minio version used. Now that they've updated the aws-sdk we just require to use the newer version. This exact same issue happened in the past with other jobs, basically AWS shipped a breaking change. I think for this one is enough with updating the version.

I do agree that we could (even should) migrate away from MinIO as the are not supporting their OSS offering anymore:
minio/minio@27742d4

Should we create a different issue for moving away from MinIO?

@raulcd
Copy link
Copy Markdown
Member

raulcd commented May 6, 2026

oh! there's an issue already, yeah, I agree we should discuss there. Thanks @tadeja !

@github-actions github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels May 6, 2026
@tadeja
Copy link
Copy Markdown
Contributor Author

tadeja commented May 6, 2026

@tadeja tadeja marked this pull request as ready for review May 6, 2026 12:06
@tadeja tadeja requested review from assignUser, jonkeane and kou as code owners May 6, 2026 12:06
Copy link
Copy Markdown
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tadeja I am unsure I understand why aws-sdk requires pinning due to the failure as the failure is unrelated to AWS (it's on arrow JSON). Is it because of the other pins?

Can you try removing the pins (maybe temporarily) to validate the S3 issue is solved? Based on the assessment only the arrow-json failure should be raised now.

@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels May 6, 2026
@github-actions github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting change review Awaiting change review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants