Batch size is too large, size must be less than or equal to: [10000]

[ZakariaHili]

Hi,
Is there an example to make a request with scroll batch because when i try to retrieve all indexed data, i receive the error :

Exception in thread "main" org.scalatest.exceptions.TestFailedException: The future returned an exception of type: com.sumologic.elasticsearch.restlastic.RestlasticSearchClient$ReturnTypes$ElasticErrorResponse, with message: ElasticsearchError(status=500): JString({"error":{"root_cause":[{"type":"query_phase_execution_exception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [10029]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"materials","node":"zGodcjEaSrCXFO9_SJuvzg","reason":{"type":"query_phase_execution_exception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [10029]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}}],"caused_by":{"type":"query_phase_execution_exception","reason":"Batch size is too large, size must be less than or equal to: [10000] but was [10029]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max_result_window] index level setting."}},"status":500}).

here my code

  var resFutQuery = restClient.query(index, tpe, new QueryRoot(MatchAll))


      var total = 10
      whenReady(resFutQuery , timeout(Span(10, Minutes))) { res =>
        total = res.rawSearchResponse.hits.total;
      }
    
    val resFutScroll = restClient.startScrollRequest(index, tpe, new QueryRoot(MatchAll, sizeOpt = Some(total)))
    whenReady(resFutScroll, timeout(Span(10, Minutes))) { res =>
      res._2.rawSearchResponse.hits.hits.foreach(
        x => {

        Process
        }
      )
    }

Thank you in advance

[CCheSumo]

@ZakariaHili

Can you try

restClient.startScrollRequest(index, tpe, new QueryRoot(MatchAll), sizeOpt= Some(total))

instead of

restClient.startScrollRequest(index, tpe, new QueryRoot(MatchAll, sizeOpt = Some(total)))

Not sure what version are you running. It looks like there may be some version incompatibilities in elasticsearch.

[ZakariaHili]

Hi
thank you for your reply,
im using "com.sumologic.elasticsearch" % "elasticsearch-aws" % "2.0.6" and 5.1 for elasticsearch
I tried you solution but it still not working

[ZakariaHili]

I think that i m not using the best way to make scoll request and there is no documentation how to used it

[CCheSumo]

Ah, the client currently supports 2.x elasticsearch. We are moving to 6.1 at the moment. Do you plan to migrate to 6.1 version in the near future?

[ZakariaHili]

I tried Elasticsearch 6.1 and its not working at all( i can't send requests to ES )
I receive the error

Exception in thread "main" org.scalatest.exceptions.TestFailedException: The future returned an exception of type: com.sumologic.elasticsearch.restlastic.RestlasticSearchClient$ReturnTypes$ElasticErrorResponse, with message: ElasticsearchError(status=406): JString({"error":"Content-Type header [text/plain; charset=UTF-8] is not supported","status":406}).
	at org.scalatest.concurrent.Futures$FutureConcept$class.tryTryAgain$1(Futures.scala:531)
	at org.scalatest.concurrent.Futures$FutureConcept$class.futureValue(Futures.scala:558)
	at org.scalatest.concurrent.ScalaFutures$$anon$1.futureValue(ScalaFutures.scala:74)
	at org.scalatest.concurrent.Futures$class.whenReady(Futures.scala:626)
	at org.scalatest.concurrent.ScalaFutures$.whenReady(ScalaFutures.scala:98)
	at services.Indexer.getAllIndex(Indexer.scala:25)
	at controller.main$.main(Main.scala:47)
	at controller.main.main(Main.scala)

[masonkirchner]

@ZakariaHili Did you ever resolve this error?

ElasticsearchError(status=406): JString({"error":"Content-Type header [text/plain; charset=UTF-8] is not supported","status":406

I am seeing it as well