Connection aborted.', RemoteDisconnected('Remote end closed connection without response') #855
Comments
|
Hi Folks, we met a similar error in several services once every several hours randomly: snuba-consumer, snuba-outcomes-consumer. Sample Exception Stacktrace: 2021-09-04 21:23:04,560 Caught ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), shutting down... During handling of the above exception, another exception occurred: Traceback (most recent call last): |
|
These look related to Clickhouse. Can you share your clickhouse logs? |
|
We're facing the same error, here are the logs of And I'm not seeing anything not normal in P.S: I didn't know what do those ids mean, so I removed first part of them, please tell me if they are unnecessary to troubleshoot. |
|
@lynnagara looks like this is stemming from arroyo. Could this be an oversight in connection pooling and handling of stale connections? |
|
Any news on this? We are facing the same exact issue as @aminvakil . I'll leave some output of |
|
We had the same issue running Sentry in Kubernetes, using As per @BYK #855 (comment) we looked more into the connectivity to clickhouse and explored the Increasing the |
|
Increasing the keep-alive-timeout from 10 to 120 seconds fixed the issue for us |
|
I have this problem: (0, "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))") how fix it ? |
|
@waxilu did you try increasing the keep-alive timeout as was suggested? You can tweak it in your clickhouse server settings |
|
This issue has gone three weeks without activity. In another week, I will close it. But! If you comment or otherwise update it, I will reset the clock, and if you remove the label "A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀 |
Snuba service has been restarted recently. The error message is as follows:
`2020-03-30 10:44:48,315 Worker flush took 21ms
2020-03-30 10:44:51,315 Flushing 3 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1437250, hi=1437252)}): forced:False size:False time:True
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
Traceback (most recent call last):
File "", line 2, in raise_from
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
six.raise_from(e, None)
File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
rv = real_getresponse(self, *args, **kwargs)
File "/usr/local/lib/python3.7/http/client.py", line 1344, in getresponse
http.client.RemoteDisconnected: Remote end closed connection without response
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
File "/usr/local/lib/python3.7/http/client.py", line 275, in _read_status
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in call
raise RemoteDisconnected("Remote end closed connection without"
During handling of the above exception, another exception occurred:
File "/usr/local/bin/snuba", line 11, in
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
return _process_result(sub_ctx.command.invoke(sub_ctx))
return ctx.invoke(self.callback, **ctx.params)
return callback(*args, **kwargs)
consumer.run()
File "/usr/src/snuba/snuba/cli/consumer.py", line 156, in consumer
File "/usr/src/snuba/snuba/utils/streams/batching.py", line 137, in run
File "/usr/src/snuba/snuba/utils/streams/batching.py", line 142, in _run_once
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
self._run_once()
rv = self.invoke(ctx)
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
self._flush()
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
File "/usr/src/snuba/snuba/consumer.py", line 98, in flush_batch
self.__writer.write(inserts)
chunked=True,
self.worker.flush_batch(self.__batch_results)
_stacktrace=sys.exc_info()[2])
File "/usr/src/snuba/snuba/utils/streams/batching.py", line 242, in _flush
File "/usr/src/snuba/snuba/clickhouse/http.py", line 73, in write
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 383, in _make_request
File "/usr/local/lib/python3.7/http/client.py", line 1344, in getresponse
File "", line 2, in raise_from
rv = real_getresponse(self, *args, **kwargs)
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
File "/usr/local/lib/python3.7/http/client.py", line 275, in _read_status
six.raise_from(e, None)
version, status, reason = self._read_status()
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse
response.begin()
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
raise RemoteDisconnected("Remote end closed connection without"`
This is because there are too many connections or the network connection is broken?
The text was updated successfully, but these errors were encountered: