Skip to content

ES|QL query builder #2997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
236 changes: 236 additions & 0 deletions docs/reference/esql-query-builder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,236 @@
# ES|QL Query Builder

The ES|QL Query Builder allows you to construct ES|QL queries using Python syntax. Consider the following example:

```python
>>> from elasticsearch.esql import ESQL
>>> query = (
ESQL.from_("employees")
.sort("emp_no")
.keep("first_name", "last_name", "height")
.eval(height_feet="height * 3.281", height_cm="height * 100")
.limit(3)
)
```

You can then see the assembled ES|QL query by printing the resulting query object:

```python
>>> query
FROM employees
| SORT emp_no
| KEEP first_name, last_name, height
| EVAL height_feet = height * 3.281, height_cm = height * 100
| LIMIT 3
```

To execute this query, you can cast it to a string and pass the string to the `client.esql.query()` endpoint:

```python
>>> from elasticsearch import Elasticsearch
>>> client = Elasticsearch(hosts=[os.environ['ELASTICSEARCH_URL']])
>>> response = client.esql.query(query=str(query))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what your thoughts are about being able to pass the query object directly here and have the conversion to string in the generated method. Maybe something we can think about for the future as an improvement.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea, and I don't think this can be a problem in the future

```

The response body contains a `columns` attribute with the list of columns included in the results, and a `values` attribute with the list of results for the query, each given as a list of column values. Here is a possible response body returned by the example query given above:

```python
>>> from pprint import pprint
>>> pprint(response.body)
{'columns': [{'name': 'first_name', 'type': 'text'},
{'name': 'last_name', 'type': 'text'},
{'name': 'height', 'type': 'double'},
{'name': 'height_feet', 'type': 'double'},
{'name': 'height_cm', 'type': 'double'}],
'is_partial': False,
'took': 11,
'values': [['Adrian', 'Wells', 2.424, 7.953144, 242.4],
['Aaron', 'Gonzalez', 1.584, 5.1971, 158.4],
['Miranda', 'Kramer', 1.55, 5.08555, 155]]}
```

## Creating an ES|QL query

To construct an ES|QL query you start from one of the ES|QL source commands:

### `ESQL.from_`

The `FROM` command selects the indices, data streams or aliases to be queried.

Examples:

```python
from elasticsearch.esql import ESQL

# FROM employees
query1 = ESQL.from_("employees")

# FROM <logs-{now/d}>
query2 = ESQL.from_("<logs-{now/d}>")

# FROM employees-00001, other-employees-*
query3 = ESQL.from_("employees-00001", "other-employees-*")

# FROM cluster_one:employees-00001, cluster_two:other-employees-*
query4 = ESQL.from_("cluster_one:employees-00001", "cluster_two:other-employees-*")

# FROM employees METADATA _id
query5 = ESQL.from_("employees").metadata("_id")
```

Note how in the last example the optional `METADATA` clause of the `FROM` command is added as a chained method.

### `ESQL.row`

The `ROW` command produces a row with one or more columns, with the values that you specify.

Examples:

```python
from elasticsearch.esql import ESQL, functions

# ROW a = 1, b = "two", c = null
query1 = ESQL.row(a=1, b="two", c=None)

# ROW a = [1, 2]
query2 = ESQL.row(a=[1, 2])

# ROW a = ROUND(1.23, 0)
query3 = ESQL.row(a=functions.round(1.23, 0))
```

### `ESQL.show`

The `SHOW` command returns information about the deployment and its capabilities.

Example:

```python
from elasticsearch.esql import ESQL

# SHOW INFO
query = ESQL.show("INFO")
```

## Adding processing commands

Once you have a query object, you can add one or more processing commands to it. The following
example shows how to create a query that uses the `WHERE` and `LIMIT` commands to filter the
results:

```python
from elasticsearch.esql import ESQL

# FROM employees
# | WHERE still_hired == true
# | LIMIT 10
query = ESQL.from_("employees").where("still_hired == true").limit(10)
```

For a complete list of available commands, review the methods of the [`ESQLBase` class](https://elasticsearch-py.readthedocs.io/en/stable/esql.html) in the Elasticsearch Python API documentation.

## Creating ES|QL Expressions and Conditions

The ES|QL query builder for Python provides two ways to create expressions and conditions in ES|QL queries.

The simplest option is to provide all ES|QL expressions and conditionals as strings. The following example uses this approach to add two calculated columns to the results using the `EVAL` command:

```python
from elasticsearch.esql import ESQL

# FROM employees
# | SORT emp_no
# | KEEP first_name, last_name, height
# | EVAL height_feet = height * 3.281, height_cm = height * 100
query = (
ESQL.from_("employees")
.sort("emp_no")
.keep("first_name", "last_name", "height")
.eval(height_feet="height * 3.281", height_cm="height * 100")
)
```

A more advanced alternative is to replace the strings with Python expressions, which are automatically translated to ES|QL when the query object is rendered to a string. The following example is functionally equivalent to the one above:

```python
from elasticsearch.esql import ESQL, E

# FROM employees
# | SORT emp_no
# | KEEP first_name, last_name, height
# | EVAL height_feet = height * 3.281, height_cm = height * 100
query = (
ESQL.from_("employees")
.sort("emp_no")
.keep("first_name", "last_name", "height")
.eval(height_feet=E("height") * 3.281, height_cm=E("height") * 100)
)
```

Here the `E()` helper function is used as a wrapper to the column name that initiates an ES|QL expression. The `E()` function transforms the given column into an ES|QL expression that can be modified with Python operators.

Here is a second example, which uses a conditional expression in the `WHERE` command:

```python
from elasticsearch.esql import ESQL

# FROM employees
# | KEEP first_name, last_name, height
# | WHERE first_name == "Larry"
query = (
ESQL.from_("employees")
.keep("first_name", "last_name", "height")
.where('first_name == "Larry"')
)
```

Using Python syntax, the condition can be rewritten as follows:

```python
from elasticsearch.esql import ESQL, E

# FROM employees
# | KEEP first_name, last_name, height
# | WHERE first_name == "Larry"
query = (
ESQL.from_("employees")
.keep("first_name", "last_name", "height")
.where(E("first_name") == "Larry")
)
```

## Using ES|QL functions

The ES|QL language includes a rich set of functions that can be used in expressions and conditionals. These can be included in expressions given as strings, as shown in the example below:

```python
from elasticsearch.esql import ESQL

# FROM employees
# | KEEP first_name, last_name, height
# | WHERE LENGTH(first_name) < 4"
query = (
ESQL.from_("employees")
.keep("first_name", "last_name", "height")
.where("LENGTH(first_name) < 4")
)
```

All available ES|QL functions have Python wrappers in the `elasticsearch.esql.functions` module, which can be used when building expressions using Python syntax. Below is the example above coded using Python syntax:

```python
from elasticsearch.esql import ESQL, functions

# FROM employees
# | KEEP first_name, last_name, height
# | WHERE LENGTH(first_name) < 4"
query = (
ESQL.from_("employees")
.keep("first_name", "last_name", "height")
.where(functions.length(E("first_name")) < 4)
)
```

Note that arguments passed to functions are assumed to be literals. When passing field names, it is necessary to wrap them with the `E()` helper function so that they are interpreted correctly.

You can find the complete list of available functions in the [ES|QL API reference documentation](https://elasticsearch-py.readthedocs.io/en/stable/esql.html#module-elasticsearch.esql.functions).
1 change: 1 addition & 0 deletions docs/reference/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ toc:
- file: connecting.md
- file: configuration.md
- file: querying.md
- file: esql-query-builder.md
- file: async.md
- file: integrations.md
children:
Expand Down
100 changes: 100 additions & 0 deletions docs/sphinx/esql.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
ES|QL Query Builder
===================

Commands
--------

.. autoclass:: elasticsearch.esql.ESQL
:inherited-members:
:members:

.. autoclass:: elasticsearch.esql.esql.ESQLBase
:inherited-members:
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.From
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Row
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Show
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.ChangePoint
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Completion
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Dissect
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Drop
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Enrich
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Eval
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Fork
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Grok
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Keep
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Limit
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.LookupJoin
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.MvExpand
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Rename
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Sample
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Sort
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Stats
:members:
:exclude-members: __init__

.. autoclass:: elasticsearch.esql.esql.Where
:members:
:exclude-members: __init__

Functions
---------

.. automodule:: elasticsearch.esql.functions
:members:
1 change: 1 addition & 0 deletions docs/sphinx/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ High-level documentation for this client is `also available <https://www.elastic
:maxdepth: 2

es_api
esql
dsl
api_helpers
exceptions
Expand Down
3 changes: 2 additions & 1 deletion elasticsearch/dsl/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
from .aggs import A, Agg
from .analysis import analyzer, char_filter, normalizer, token_filter, tokenizer
from .document import AsyncDocument, Document
from .document_base import InnerDoc, M, MetaField, mapped_field
from .document_base import E, InnerDoc, M, MetaField, mapped_field
from .exceptions import (
ElasticsearchDslException,
IllegalOperation,
Expand Down Expand Up @@ -135,6 +135,7 @@
"Double",
"DoubleRange",
"DslBase",
"E",
"ElasticsearchDslException",
"EmptySearch",
"Facet",
Expand Down
Loading
Loading