Concurrency issue leading to deadlock on produce

[erenboz]

There seems to be continuation issue to #777 where fix doesn't seem completely concurrency correct.

We have hit this bug franz-go leaking whole lot of goroutines where we call produce without cancel from a spawned goroutine for each. Initially thought it was bad concurrency around wait condition but it doesn't seem to be the case I just missed Cond was reusing same mutex.

We hit this issue on highly parallelized usage where we had gomaxprocs > 45 on 36 physical core cpu so might require just right kind of race to reproduce in test. If change is not acceptable as is with current test, I might look into writing one later.

[erenboz]

We'll try to set something reproducing in prod to find whereabouts

Maxbuffered records set at 5, GOMAXPROCS 80 running on 80 CPU gke n2 node stack looks like

No kafka errors, on callbacks, median latency over 10s (metric bucket limitation). Steady climb on goroutines indicating Some produce calls are not returning.

Our effective code is 6-7x per request (300-400 request/sec) on 5-6 different topics, partitions up to 960, max message bytes up to 4M

go kclient.Produce(
		context.Background(),
  		&kgo.Record{
  			Value: message,
  			Topic: topic,
  		}, callback))

[twmb]

What's the callback? What's the stack trace once you're at that many goroutines? (you likely need to redact a lot in the stack trace).

Not to say the code in franz-go is flawless -- it definitely isn't (hence the dozens of patch releases...) -- but so far every report of a goroutine leak in the past few years has been a bug in the usage around the client.

[erenboz]

Yeah that's fair, I tried writing a test within library codebase but was unable to reproduce anything. There is nothing obvious in callback that'd block or leak goroutines but I'll give it a try to put similar test in the service for isolation when back to work.