Skip to content

ES May Permanently Block During Concurrent Query Operations #963

Closed
@EmptyDreams

Description

@EmptyDreams

Java API client version

8.17.0

Java version

21

Elasticsearch Version

8.10.2

Problem description

When I use concurrent requests to ES with the code below, the ES Java API may permanently block the current thread, eventually depleting my thread pool resources and causing my program to completely freeze.

My code is:

@JvmStatic
internal val client = run {
    RestClient.builder(HttpHost.create(LogScannerMain.main.pluginConfig.esHost))
        .setCompressionEnabled(true)
        .build()
}

@JvmStatic
private suspend fun postSearch(request: SearchRequest, view: String): EsResponse {
    val response = withContext(Dispatchers.IO) {
        suspendCoroutine {
            client.performRequestAsync(Request("POST", "/$view/_search").apply {
                val jsonText = FastJsonProvider.createGenerator().use { jsonGenerator ->
                    request.serialize(jsonGenerator, FastJsonMapper())
                    jsonGenerator.toString()
                }
                LogScannerMain.main.logger.trace { "post search: $jsonText" }
                entity = NStringEntity(jsonText, ContentType.APPLICATION_JSON)
            }, SuspendResponseListener(it))
        }
    }
    val text = EntityUtils.toString(response.entity)
    return JSON.parseObject(text, EsResponse::class.java)
}

When ES blocks, the ES-related dump content obtained from jstack is as follows:

"elasticsearch-rest-client-0-thread-1" #48 [8978] daemon prio=5 os_prio=0 cpu=90.04ms elapsed=1506.58s tid=0x0000ffff701390f0 nid=8978 runnable  [0x0000ffff42dde000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait([email protected]/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:121)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:130)
        - locked <0x00000006c17fcae8> (a sun.nio.ch.Util$2)
        - locked <0x00000006c17fca98> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:142)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:343)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.lang.Thread.runWith([email protected]/Thread.java:1596)
        at java.lang.Thread.run([email protected]/Thread.java:1583)

"elasticsearch-rest-client-0-thread-2" #49 [8979] daemon prio=5 os_prio=0 cpu=145.01ms elapsed=1506.57s tid=0x0000ffffa80ea4c0 nid=8979 runnable  [0x0000ffff42be0000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait([email protected]/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:121)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:130)
        - locked <0x00000006c18676a0> (a sun.nio.ch.Util$2)
        - locked <0x00000006c17fcef0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:142)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.lang.Thread.runWith([email protected]/Thread.java:1596)
        at java.lang.Thread.run([email protected]/Thread.java:1583)

"elasticsearch-rest-client-0-thread-3" #50 [8980] daemon prio=5 os_prio=0 cpu=192.02ms elapsed=1506.57s tid=0x0000ffffa82d17c0 nid=8980 runnable  [0x0000ffff429e2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait([email protected]/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:121)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:130)
        - locked <0x00000006c1882158> (a sun.nio.ch.Util$2)
        - locked <0x00000006c186f938> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:142)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.lang.Thread.runWith([email protected]/Thread.java:1596)
        at java.lang.Thread.run([email protected]/Thread.java:1583)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions