Skip to content

ES|QL query builder #2997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions docs/reference/esql-query-builder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# ES|QL Query Builder

The ES|QL Query Builder allows you to construct ES|QL queries using Python syntax. Consider the following example:

```python
>>> from elasticsearch.esql import ESQL
>>> query = (
ESQL.from_("employees")
.sort("emp_no")
.keep("first_name", "last_name", "height")
.eval(height_feet="height * 3.281", height_cm="height * 100")
)
```

You can then see the assembled ES|QL query by printing the resulting query object:

```python
>>> query
FROM employees
| SORT emp_no
| KEEP first_name, last_name, height
| EVAL height_feet = height * 3.281, height_cm = height * 100
```

To execute this query, you can cast it to a string and pass the string to the `client.esql.query()` endpoint:

```python
>>> response = client.esql.query(query=str(query))
```

The response containts a `columns` attribute with the list of columns included in each result, and a `values` attribute with the list of results for the query.

## Creating an ES|QL query

To construct an ES|QL query you start from one of the ES|QL source commands:

### `ESQL.from_`

The `FROM` command selects the indices, data streams or aliases to be queried.

Examples:

```python
from elasticsearch.esql import ESQL

query1 = ESQL.from_("employees")
query2 = ESQL.from_("<logs-{now/d}>")
query3 = ESQL.from_("employees-00001", "other-employees-*")
query4 = ESQL.from_("cluster_one:employees-00001", "cluster_two:other-employees-*")
query5 = ESQL.from_("employees").metadata("_id")
```

Note how in the last example the optional

### `ESQL.row`

The `ROW` command produces a row with one or more columns, with the values that you specify.

Examples:

```python
from elasticsearch.esql import ESQL, functions

query1 = ESQL.row(a=1, b="two", c=None)
query2 = ESQL.row(a=[1, 2])
query3 = ESQL.row(a=functions.round(1.23, 0))
```

### `ESQL.show`

The `SHOW` command returns information about the deployment and its capabilities.

Example:

```python
query = ESQL.show("INFO")
```

## Adding processing commands

Once you have a query object, you can start adding processing commands to it. The following
example shows how to create a query that uses the `WHERE` and `LIMIT` clauses to filter the
results:

```python
query = ESQL.from_("employees").where("still_hired == true").limit(10)
```

For a complete list of available commands, review the methods of the [`ESQLBase` class](https://elasticsearch-py.readthedocs.io/en/stable/esql.html) in the Elasticsearch Python API documentation.

## Creating Expressions and Conditions

The ES|QL query builder for Python provides two different ways to create expressions and conditions in ES|QL queries.

The most basic option is to provide expressions as strings. Consider the following example, which defines two expressions in the `EVAL` command:

```python
query = (
ESQL.from_("employees")
.sort("emp_no")
.keep("first_name", "last_name", "height")
.eval(height_feet="height * 3.281", height_cm="height * 100")
)
```

A more advanced alternative is to use Python expressions, which are automatically translated to ES|QL. The following example is functionally equivalent to the above:

```python
from elasticsearch.esql import E

query = (
ESQL.from_("employees")
.sort("emp_no")
.keep("first_name", "last_name", "height")
.eval(height_feet=E("height") * 3.281, height_cm=E("height") * 100)
)
```

Here the `E()` helper function is used as a wrapper that accepts a column name and transforms it into an ES|QL expression that can be modified with Python operations.

Here is a second example, which uses a function and a comparison in the `WHERE` command:

```python
query = (
ESQL.from_("employees")
.keep("first_name", "last_name", "height")
.where("LENGTH(first_name) < 4")
)
```

Using Python syntax, the condition can be rewritten as follows:

```python
from elasticsearch.esql import E, functions

query = (
ESQL.from_("employees")
.keep("first_name", "last_name", "height")
.where(functions.length("first_name") < 4)
)
```

All available ES|QL functions have Python wrappers in the `esql.functions` module. Because the functions already return ES|QL expressions, it is not necessary to wrap them with the `E()` helper function, which is only used when starting an expression from a field name given as a string.

You can find the complete list of available functions in the [API reference documentation](https://elasticsearch-py.readthedocs.io/en/stable/esql.html#module-elasticsearch.esql.functions).
1 change: 1 addition & 0 deletions docs/reference/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ toc:
- file: connecting.md
- file: configuration.md
- file: querying.md
- file: esql-query-builder.md
- file: async.md
- file: integrations.md
children:
Expand Down
92 changes: 92 additions & 0 deletions docs/sphinx/esql.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
ES|QL Query Builder
===================

Commands
--------

.. autoclass:: elasticsearch.esql.ESQL
:inherited-members:
:members:

.. autoclass:: elasticsearch.esql.esql.ESQLBase
:inherited-members:
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.From
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Row
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Show
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Dissect
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Drop
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Enrich
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Eval
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Fork
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Grok
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Keep
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Limit
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.LookupJoin
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.MvExpand
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Rename
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Sample
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Sort
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Stats
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Where
:members:
:exclude-members: __init__

Functions
---------

.. automodule:: elasticsearch.esql.functions
:members:
1 change: 1 addition & 0 deletions docs/sphinx/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ High-level documentation for this client is `also available <https://www.elastic
:maxdepth: 2

es_api
esql
dsl
api_helpers
exceptions
Expand Down
3 changes: 2 additions & 1 deletion elasticsearch/dsl/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
from .aggs import A, Agg
from .analysis import analyzer, char_filter, normalizer, token_filter, tokenizer
from .document import AsyncDocument, Document
from .document_base import InnerDoc, M, MetaField, mapped_field
from .document_base import E, InnerDoc, M, MetaField, mapped_field
from .exceptions import (
ElasticsearchDslException,
IllegalOperation,
Expand Down Expand Up @@ -135,6 +135,7 @@
"Double",
"DoubleRange",
"DslBase",
"E",
"ElasticsearchDslException",
"EmptySearch",
"Facet",
Expand Down
Loading
Loading