压测aeraki.meta_protocol.filters.istio_stats会导致envoy发生crash

[wuyouxia123]

测试步骤：

1. 测试demo
   
2. jemter 并发数测试为500进行压测
   
3. 服务端istio-proxy会出现crash
4. 删除envoyfilter中的aeraki.meta_protocol.filters.istio_stats配置，重新压测，压测通过
   

出错log：
2023-09-21T07:56:39.443577Z critical envoy assert external/envoy/source/common/stats/symbol_table.cc:497 assert failure: search != decode_map_.end(). Details: no such symbol thread=30
2023-09-21T07:56:39.443887Z critical envoy backtrace external/envoy/source/server/backtrace.h:104 Caught Aborted, suspect faulting address 0x53900000014 thread=30
2023-09-21T07:56:39.443935Z critical envoy backtrace external/envoy/source/server/backtrace.h:91 Backtrace (use tools/stack_decode.py to get line numbers): thread=30
2023-09-21T07:56:39.443949Z critical envoy backtrace external/envoy/source/server/backtrace.h:92 Envoy version: 8d280ad/1.26.4/Modified/DEBUG/BoringSSL thread=30
2023-09-21T07:56:39.739249Z critical envoy assert external/envoy/source/common/stats/symbol_table.cc:688 assert failure: mem_block.capacityRemaining() == 0. thread=32

[zhaohuabing]

测试环境中 accessLogFormat 配置贴出来看一下呢

[wuyouxia123]

测试环境中 accessLogFormat 配置贴出来看一下呢

没有配置accesslog，使用istio默认的accesslog

[zhaohuabing]

使用这个试试

https://github.com/aeraki-mesh/aeraki/blob/25c89759986ef7858af011b1c3ead87b6e5f0571/test/e2e/common/istio-config.yaml#L17

accessLogFormat: "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)%
     %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\"
     %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER%
     %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"

[wuyouxia123]

使用这个试试

https://github.com/aeraki-mesh/aeraki/blob/25c89759986ef7858af011b1c3ead87b6e5f0571/test/e2e/common/istio-config.yaml#L17

accessLogFormat: "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)%
     %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\"
     %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER%
     %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"

配置accesslog后还是有同样的问题和内存不足的问题

[zhaohuabing]

Envoy stats 是比较耗内存的，你把内存调大试一下。

[wuyouxia123]

提高内存仍然有同样的问题，且压测时未看到内存有明显提升，且未超过分配值

1. 内存分配
   

2. 调整测试demo为并发2次，持续300s，会压测一段时间后发生crash
   

3. envoy占用内存未见明显提升
   客户端envoy pid：3348825
   服务端envoy pid：3346735
   
   

[zhaohuabing]

@wuyouxia123 请问这个问题解决了吗？

[wuyouxia123]

没有

[yizhengx]

同样出现了这个crash以及OOM在1.4.2 version 用wrk2压测meta_thrift demo中，请问有没有办法解决呢？