mimirtool backfill - compactor: can't check block state/block validation failed

[litetex]

Describe the bug

When running mimirtool backfill I get caller=block_upload.go:207 level=warn component=compactor user=tdr feature="block upload" block=01J4WNXPS4GSR8AMK9A4732704 operation="start block upload" request_id=17ec82ff283a34ff msg="can't check block state" response="block validation failed" status=400

Setting the log level to debug doesn't help at all.

It would be nice to know WHY the validation failed so that I can diagnose and address the problem...

To Reproduce

Sorry but I can't paste my complete setup here.

Basically:

1. Start mimir
2. Run backfill for prometheus:
   mimirtool backfill --address=http://mimir-backend:9009 --id=<redacted> /data/prometheus/01J4WNXPS4GSR8AMK9A4732704 ...

Expected behavior

See an error message that says what caused the block validation error so that I can fix it.

Environment

irrelevant
Mimir version: 2.13.0

Additional Context

Same problem was already encountered in #8164 but no efforts were made to improve the situation :/

Likely these pieces of code should rethrow the error and not just "swallow" it:

mimir/pkg/compactor/block_upload.go

Lines 766 to 768 in 3872ccb

	if v.Error != "" {
	return blockValidationFailed, meta, v, err
	}

mimir/pkg/compactor/block_upload.go

Lines 733 to 735 in 3872ccb

	case blockValidationFailed:
	return m, v, httpError{message: "block validation failed", statusCode: http.StatusBadRequest}
	}

[litetex]

Also note that compactor_block_upload_validation_enabled: false doesn't help... The error is still the same