Test flake: TestDistributor_Push_ShouldSupportWriteBothToIngestersAndPartitions

[seizethedave]

Describe the bug

Received a test flake in CI: https://github.com/grafana/mimir/actions/runs/10856690387/job/30131835759#step:8:136

--- FAIL: TestDistributor_Push_ShouldSupportWriteBothToIngestersAndPartitions (0.00s)
    --- FAIL: TestDistributor_Push_ShouldSupportWriteBothToIngestersAndPartitions/should_shard_series_across_all_partitions_when_shuffle_sharding_is_disabled (3.00s)
        distributor_ingest_storage_test.go:455: 
            	Error Trace:	/__w/mimir/mimir/pkg/distributor/distributor_ingest_storage_test.go:455
            	Error:      	Not equal: 
            	            	expected: map[int32][]string{0:[]string{"series_four", "series_one", "series_three"}, 1:[]string{"series_two"}, 2:[]string{"series_five"}}
            	            	actual  : map[int32][]string{2:[]string{"series_five"}}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,10 +1,2 @@
            	            	-(map[int32][]string) (len=3) {
            	            	- (int32) 0: ([]string) (len=3) {
            	            	-  (string) (len=11) "series_four",
            	            	-  (string) (len=10) "series_one",
            	            	-  (string) (len=12) "series_three"
            	            	- },
            	            	- (int32) 1: ([]string) (len=1) {
            	            	-  (string) (len=10) "series_two"
            	            	- },
            	            	+(map[int32][]string) (len=1) {
            	            	  (int32) 2: ([]string) (len=1) {
            	Test:       	TestDistributor_Push_ShouldSupportWriteBothToIngestersAndPartitions/should_shard_series_across_all_partitions_when_shuffle_sharding_is_disabled

Environment

Github actions CI.

[dimitarvdimitrov]

i couldn't reproduce after about 600 runs.

After reducing the time to poll from kafka from 1s to 10ms I could get a similar failure: although there it's missing all 3 partitions, not just 2/3.

diff --git a/pkg/distributor/distributor_ingest_storage_test.go b/pkg/distributor/distributor_ingest_storage_test.go
index b598ac6a8f..0ac85ac77a 100644
--- a/pkg/distributor/distributor_ingest_storage_test.go
+++ b/pkg/distributor/distributor_ingest_storage_test.go
@@ -451,7 +451,7 @@ func TestDistributor_Push_ShouldSupportWriteBothToIngestersAndPartitions(t *test
 			require.NoError(t, err)
 
 			// Ensure series has been correctly sharded to partitions.
-			actualSeriesByPartition := readAllMetricNamesByPartitionFromKafka(t, kafkaCluster.ListenAddrs(), testConfig.ingestStoragePartitions, time.Second)
+			actualSeriesByPartition := readAllMetricNamesByPartitionFromKafka(t, kafkaCluster.ListenAddrs(), testConfig.ingestStoragePartitions, time.Second/100)
 			assert.Equal(t, testData.expectedMetricsByPartition, actualSeriesByPartition)
 
 			// Ensure series have been correctly sharded to ingesters.

Failure

    distributor_ingest_storage_test.go:455: 
        	Error Trace:	/Users/dimitar/grafana/mimir/pkg/distributor/distributor_ingest_storage_test.go:455
        	Error:      	Not equal: 
        	            	expected: map[int32][]string{0:[]string{"series_four", "series_one", "series_three"}, 1:[]string{"series_two"}, 2:[]string{"series_five"}}
        	            	actual  : map[int32][]string{}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,13 +1,2 @@
        	            	-(map[int32][]string) (len=3) {
        	            	- (int32) 0: ([]string) (len=3) {
        	            	-  (string) (len=11) "series_four",
        	            	-  (string) (len=10) "series_one",
        	            	-  (string) (len=12) "series_three"
        	            	- },
        	            	- (int32) 1: ([]string) (len=1) {
        	            	-  (string) (len=10) "series_two"
        	            	- },
        	            	- (int32) 2: ([]string) (len=1) {
        	            	-  (string) (len=11) "series_five"
        	            	- }
        	            	+(map[int32][]string) {
        	            	 }

I opened a PR to log the end offsets of the topic on a failure: #9323 so we can verify if the timeout is just too tight