Simplify fuzzing coverage (CI) scripts somewhat #3925

TheBlueMatt · 2025-07-14T14:59:21Z

We recently introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. Here, we simplify some of the scripts that do so, especially removing the second ci-fuzz.sh file we added to the repo.

ldk-reviews-bot · 2025-07-14T14:59:24Z

👋 Thanks for assigning @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

graphite-app · 2025-07-14T15:01:13Z

.github/workflows/build.yml

+          ./contrib/generate_fuzz_coverage.sh --output-dir `pwd`
+          # Could you use this to fake the coverage report for your PR? Sure.
+          # Will anyone be impressed by your amazing coverage? No
+          # Maybe if codecov wasn't broken we wouldn't need to do this...
+          bash <(curl -s https://codecov.io/bash) -f fuzz-codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57"


There appears to be a path mismatch in the coverage file handling. The script is called with --output-dir \pwd`which will create the coverage file at`pwd`/fuzz-codecov.json, but the codecov upload command is looking for fuzz-codecov.json` in the current directory without a path prefix.

To fix this, either:

Add the --output-codecov-json flag to the script call, or

Update the file path in the bash command to \pwd`/fuzz-codecov.json`

This will ensure the codecov upload can find the generated coverage file.

Suggested change

./contrib/generate_fuzz_coverage.sh --output-dir `pwd`

# Could you use this to fake the coverage report for your PR? Sure.

# Will anyone be impressed by your amazing coverage? No

# Maybe if codecov wasn't broken we wouldn't need to do this...

bash <(curl -s https://codecov.io/bash) -f fuzz-codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57"

./contrib/generate_fuzz_coverage.sh --output-dir `pwd`

# Could you use this to fake the coverage report for your PR? Sure.

# Will anyone be impressed by your amazing coverage? No

# Maybe if codecov wasn't broken we wouldn't need to do this...

bash <(curl -s https://codecov.io/bash) -f `pwd`/fuzz-codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57"

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

joostjager

Small PR, but still difficult to review without much explanation. Commit message just says 'simplify', but adding self-comments in the PR explaining why this is all equivalent would be helpful.

ci/ci-fuzz.sh

.github/workflows/build.yml

contrib/generate_fuzz_coverage.sh

ldk-reviews-bot · 2025-07-15T09:57:15Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

graphite-app · 2025-07-15T12:16:31Z

contrib/generate_fuzz_coverage.sh

+		--output-codecov-json)
+			OUTPUT_CODECOV_JSON=1
+			shift 1
+			;;
        *)
            echo "Unknown option: $1"
            echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY]"


The usage information is missing the newly added --output-codecov-json flag. Please update the help text to include this option:

echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY] [--output-codecov-json]"

This will help users understand all available command-line options.

Suggested change

echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY]"

echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY] [--output-codecov-json]"

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

Its a CI-only flag so not really worth documenting.

codecov · 2025-07-15T12:25:02Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.02%. Comparing base (b939100) to head (37cd32d).
Report is 83 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##             main    #3925     +/-   ##
=========================================
  Coverage   89.01%   89.02%             
=========================================
  Files         166      167      +1     
  Lines      119692   121800   +2108     
  Branches   119692   121800   +2108     
=========================================
+ Hits       106546   108427   +1881     
- Misses      10740    10956    +216     
- Partials     2406     2417     +11

Flag	Coverage Δ
fuzz	`?`
fuzzing	`22.73% <ø> (?)`
tests	`88.84% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

We recently introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. Here, we simplify some of the scripts that do so, especially removing the second `ci-fuzz.sh` file we added to the repo.

codecov recommends using the new CLI uploader rather than their (deprecated) bash uploader (which we use). It also appears to have a fail-on-error mode which we'd prefer over the bash one which silently swallows errors.

TheBlueMatt · 2025-07-18T12:12:47Z

Okay, this seems to work, and codecov's report includes coverage for lines that are #[cfg(fuzzing)].

joostjager

If I understand it correctly, all the coverage now ends up in the same report?

joostjager · 2025-07-18T13:44:14Z

.github/workflows/build.yml

          # Could you use this to fake the coverage report for your PR? Sure.
          # Will anyone be impressed by your amazing coverage? No
          # Maybe if codecov wasn't broken we wouldn't need to do this...
-          bash <(curl -s https://codecov.io/bash) -f target/codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57"
+          ./codecov --verbose upload-process --disable-search --fail-on-error -f target/codecov.json -t "f421b687-4dc2-4387-ac3d-dc3b2528af57"


From docs:

./codecov --verbose upload-process --disable-search -t ${{ secrets.CODECOV_TOKEN }} -n 'job-name'-${{ github.run_id }} -F <flag name> -f <covarage report file name>

Don't we need the job name?

And the flag, is that useful to keep apart fuzz and normal test coverage?

We can try that, indeed the current logic merges it into one total coverage report.

Not sure if separated is what we want. Perhaps the UI allows the separate parts to be merged as well, so we can choose?

Btw, I thought the -F flag was for indicating what the source of the coverage was.

Ah, indeed, I hadn't seen that. With or without the job name doesn't seem to accomplish much. Codecov automatically detects that its in a PR and what the commit hash is, so the inclusion of a name doesn't seem to change much in the UI. I pushed an update to use flags, so hopefully that lets us filter in the UI (the name doesn't seem to, despite having a checkbox beside them..)

Okay, yea, that seems to work. I had to turn on flags support on our repo, but you can now filter by the flags.

Its useful to be able to identify what kind of coverage we have for a line - test coverage is very different in nature from fuzzing coverage. Here we pass separate "job names" to codecov uploads to do so.

graphite-app bot reviewed Jul 14, 2025

View reviewed changes

ldk-reviews-bot requested a review from joostjager July 14, 2025 15:10

joostjager reviewed Jul 15, 2025

View reviewed changes

graphite-app bot reviewed Jul 15, 2025

View reviewed changes

TheBlueMatt force-pushed the 2025-07-3718-followups branch from 95df8a0 to 427d984 Compare July 16, 2025 17:30

TheBlueMatt added 2 commits July 17, 2025 19:47

Simplify fuzzing coverage (CI) scripts somewhat

4e3d54e

We recently introduced uploading coverage from no-corpus fuzzing runs into codecov in CI. Here, we simplify some of the scripts that do so, especially removing the second `ci-fuzz.sh` file we added to the repo.

Switch to codecov CLI

08795f2

codecov recommends using the new CLI uploader rather than their (deprecated) bash uploader (which we use). It also appears to have a fail-on-error mode which we'd prefer over the bash one which silently swallows errors.

TheBlueMatt force-pushed the 2025-07-3718-followups branch from 427d984 to 08795f2 Compare July 17, 2025 19:47

TheBlueMatt requested a review from joostjager July 18, 2025 12:12

joostjager reviewed Jul 18, 2025

View reviewed changes

Separate codecov reports out between testing and fuzzing

37cd32d

Its useful to be able to identify what kind of coverage we have for a line - test coverage is very different in nature from fuzzing coverage. Here we pass separate "job names" to codecov uploads to do so.

TheBlueMatt force-pushed the 2025-07-3718-followups branch from 059c0cf to 37cd32d Compare July 18, 2025 22:17

	echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY]"
	echo "Usage: $0 [--output-dir OUTPUT_DIRECTORY] [--output-codecov-json]"

Simplify fuzzing coverage (CI) scripts somewhat #3925

Are you sure you want to change the base?

Simplify fuzzing coverage (CI) scripts somewhat #3925

Conversation

TheBlueMatt commented Jul 14, 2025

Uh oh!

ldk-reviews-bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graphite-app bot Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ldk-reviews-bot commented Jul 15, 2025

Uh oh!

graphite-app bot Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheBlueMatt commented Jul 18, 2025

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

joostjager Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

joostjager Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjager Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ldk-reviews-bot commented Jul 14, 2025 •

edited

Loading

codecov bot commented Jul 15, 2025 •

edited

Loading

joostjager Jul 18, 2025 •

edited

Loading