net/http: TLS handshake timeout and net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[ramesh569]

Describe the issue you're reporting

we have deployed otel collector agent through helm chart into aks cluster. it is able to run successfully for sometime. later it was giving following error

2023-06-21T07:43:49.299Z info exporterhelper/queued_retry.go:433 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://xxxx:8088/services/collector\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "5.78910031s"}
2023-06-21T10:01:34.181Z info exporterhelper/queued_retry.go:433 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://xxxx:8088/services/collector\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)", "interval": "4.482053057s"}

then increased http timeout from 10s to 20s. then these errors got disappear, after sometime its giving below error.

2023-06-21T12:59:25.157Z info exporterhelper/queued_retry.go:433 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://xxxx:8088/services/collector\": net/http: TLS handshake timeout", "interval": "2.685233659s"}
2023-06-21T13:06:48.357Z info exporterhelper/queued_retry.go:433 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://xxxx:8088/services/collector\": net/http: TLS handshake timeout", "interval": "3.036162409s"}

we set insecureSkipVerify: true in values.yaml file, but not sure how this error is reporting. any help is much appreciated. thanks in advance

[ramesh569]

i am getting different error now.

2023-06-22T13:33:04.352Z info exporterhelper/queued_retry.go:433 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "Post "https://****:8088/services/collector": net/http: request canceled (Client.Timeout exceeded while awaiting headers)", "interval": "5.179037564s"}

Any idea why timeout exception although timeout value increased from 10s to 20s ?

[matthewmodestino]

Usually it just means the splunk endpoint could be overloaded. It may not be necessary to raise the timeout as the collector will just retry, and usually succeed. Looks like you might be sending to a single hec endpoint? you may need to tune the inputs, or resources, or look at using a load balancer and multiple endpoints depending on how many collectors you are trying to point at your hec receiver.

[ramesh569]

Thank you matthew, you are right, i am sending data to single hec endpoint and running single collector on aks cluster . is it wise to deploy collector aget as load balancer when the env is small ?. usually collector sends log files from mounted path on fileshare . increased timeout but issue still persist. could you please specify what inputs, resources needs to be tuned to avoid this ?
thanking you for valueble suggestions

[ramesh569]

Any fine tune configuration to avoid timeout issues ?

[ramesh569]

its throwing following error again.

2023-07-13T10:09:47.909Z error exporterhelper/queued_retry.go:175 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "splunk_hec/platform_logs", "error": "max elapsed time expired Post "https://xxxx:8088/services/collector\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)", "dropped_items": 2}

Any leads to avoid timeout issues?