-
Notifications
You must be signed in to change notification settings - Fork 1.2k
ES|QL query builder #2997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
miguelgrinberg
wants to merge
12
commits into
elastic:main
Choose a base branch
from
miguelgrinberg:esql-query-builder
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
ES|QL query builder #2997
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
54c7525
ES|QL query builder
miguelgrinberg ed0d0b0
add missing esql api documentation
miguelgrinberg 748e27a
add FORK command
miguelgrinberg 7f6dc00
initial attempt at generating all functions
miguelgrinberg 3d88ee5
unit tests
miguelgrinberg 27d8977
more operators
miguelgrinberg 09a1f99
documentation
miguelgrinberg c6fb3f9
integration tests
miguelgrinberg 1f783d2
add new COMPLETION command
miguelgrinberg 27bf417
show ES|QL in all docs examples
miguelgrinberg 78e50df
Merge branch 'main' into esql-query-builder
miguelgrinberg 5983fb2
Docstring fixes
miguelgrinberg File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,236 @@ | ||
# ES|QL Query Builder | ||
|
||
The ES|QL Query Builder allows you to construct ES|QL queries using Python syntax. Consider the following example: | ||
|
||
```python | ||
>>> from elasticsearch.esql import ESQL | ||
>>> query = ( | ||
ESQL.from_("employees") | ||
.sort("emp_no") | ||
.keep("first_name", "last_name", "height") | ||
.eval(height_feet="height * 3.281", height_cm="height * 100") | ||
.limit(3) | ||
) | ||
``` | ||
|
||
You can then see the assembled ES|QL query by printing the resulting query object: | ||
|
||
```python | ||
>>> query | ||
FROM employees | ||
| SORT emp_no | ||
| KEEP first_name, last_name, height | ||
| EVAL height_feet = height * 3.281, height_cm = height * 100 | ||
| LIMIT 3 | ||
``` | ||
|
||
To execute this query, you can cast it to a string and pass the string to the `client.esql.query()` endpoint: | ||
|
||
```python | ||
>>> from elasticsearch import Elasticsearch | ||
>>> client = Elasticsearch(hosts=[os.environ['ELASTICSEARCH_URL']]) | ||
>>> response = client.esql.query(query=str(query)) | ||
``` | ||
|
||
The response body contains a `columns` attribute with the list of columns included in the results, and a `values` attribute with the list of results for the query, each given as a list of column values. Here is a possible response body returned by the example query given above: | ||
|
||
```python | ||
>>> from pprint import pprint | ||
>>> pprint(response.body) | ||
{'columns': [{'name': 'first_name', 'type': 'text'}, | ||
{'name': 'last_name', 'type': 'text'}, | ||
{'name': 'height', 'type': 'double'}, | ||
{'name': 'height_feet', 'type': 'double'}, | ||
{'name': 'height_cm', 'type': 'double'}], | ||
'is_partial': False, | ||
'took': 11, | ||
'values': [['Adrian', 'Wells', 2.424, 7.953144, 242.4], | ||
['Aaron', 'Gonzalez', 1.584, 5.1971, 158.4], | ||
['Miranda', 'Kramer', 1.55, 5.08555, 155]]} | ||
``` | ||
|
||
## Creating an ES|QL query | ||
|
||
To construct an ES|QL query you start from one of the ES|QL source commands: | ||
|
||
### `ESQL.from_` | ||
|
||
The `FROM` command selects the indices, data streams or aliases to be queried. | ||
|
||
Examples: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL | ||
|
||
# FROM employees | ||
query1 = ESQL.from_("employees") | ||
|
||
# FROM <logs-{now/d}> | ||
query2 = ESQL.from_("<logs-{now/d}>") | ||
|
||
# FROM employees-00001, other-employees-* | ||
query3 = ESQL.from_("employees-00001", "other-employees-*") | ||
|
||
# FROM cluster_one:employees-00001, cluster_two:other-employees-* | ||
query4 = ESQL.from_("cluster_one:employees-00001", "cluster_two:other-employees-*") | ||
|
||
# FROM employees METADATA _id | ||
query5 = ESQL.from_("employees").metadata("_id") | ||
``` | ||
|
||
Note how in the last example the optional `METADATA` clause of the `FROM` command is added as a chained method. | ||
miguelgrinberg marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### `ESQL.row` | ||
|
||
The `ROW` command produces a row with one or more columns, with the values that you specify. | ||
|
||
Examples: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL, functions | ||
|
||
# ROW a = 1, b = "two", c = null | ||
query1 = ESQL.row(a=1, b="two", c=None) | ||
|
||
# ROW a = [1, 2] | ||
query2 = ESQL.row(a=[1, 2]) | ||
|
||
# ROW a = ROUND(1.23, 0) | ||
query3 = ESQL.row(a=functions.round(1.23, 0)) | ||
``` | ||
|
||
### `ESQL.show` | ||
|
||
The `SHOW` command returns information about the deployment and its capabilities. | ||
|
||
Example: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL | ||
|
||
# SHOW INFO | ||
query = ESQL.show("INFO") | ||
``` | ||
|
||
## Adding processing commands | ||
|
||
Once you have a query object, you can add one or more processing commands to it. The following | ||
example shows how to create a query that uses the `WHERE` and `LIMIT` commands to filter the | ||
results: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL | ||
|
||
# FROM employees | ||
# | WHERE still_hired == true | ||
# | LIMIT 10 | ||
query = ESQL.from_("employees").where("still_hired == true").limit(10) | ||
``` | ||
|
||
For a complete list of available commands, review the methods of the [`ESQLBase` class](https://elasticsearch-py.readthedocs.io/en/stable/esql.html) in the Elasticsearch Python API documentation. | ||
|
||
## Creating ES|QL Expressions and Conditions | ||
|
||
The ES|QL query builder for Python provides two ways to create expressions and conditions in ES|QL queries. | ||
|
||
The simplest option is to provide all ES|QL expressions and conditionals as strings. The following example uses this approach to add two calculated columns to the results using the `EVAL` command: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL | ||
|
||
# FROM employees | ||
# | SORT emp_no | ||
# | KEEP first_name, last_name, height | ||
# | EVAL height_feet = height * 3.281, height_cm = height * 100 | ||
query = ( | ||
ESQL.from_("employees") | ||
.sort("emp_no") | ||
.keep("first_name", "last_name", "height") | ||
.eval(height_feet="height * 3.281", height_cm="height * 100") | ||
) | ||
``` | ||
|
||
A more advanced alternative is to replace the strings with Python expressions, which are automatically translated to ES|QL when the query object is rendered to a string. The following example is functionally equivalent to the one above: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL, E | ||
|
||
# FROM employees | ||
# | SORT emp_no | ||
# | KEEP first_name, last_name, height | ||
# | EVAL height_feet = height * 3.281, height_cm = height * 100 | ||
query = ( | ||
ESQL.from_("employees") | ||
.sort("emp_no") | ||
.keep("first_name", "last_name", "height") | ||
.eval(height_feet=E("height") * 3.281, height_cm=E("height") * 100) | ||
) | ||
``` | ||
|
||
Here the `E()` helper function is used as a wrapper to the column name that initiates an ES|QL expression. The `E()` function transforms the given column into an ES|QL expression that can be modified with Python operators. | ||
|
||
Here is a second example, which uses a conditional expression in the `WHERE` command: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL | ||
|
||
# FROM employees | ||
# | KEEP first_name, last_name, height | ||
# | WHERE first_name == "Larry" | ||
query = ( | ||
ESQL.from_("employees") | ||
.keep("first_name", "last_name", "height") | ||
.where('first_name == "Larry"') | ||
) | ||
``` | ||
|
||
Using Python syntax, the condition can be rewritten as follows: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL, E | ||
|
||
# FROM employees | ||
# | KEEP first_name, last_name, height | ||
# | WHERE first_name == "Larry" | ||
query = ( | ||
ESQL.from_("employees") | ||
.keep("first_name", "last_name", "height") | ||
.where(E("first_name") == "Larry") | ||
) | ||
``` | ||
|
||
## Using ES|QL functions | ||
|
||
The ES|QL language includes a rich set of functions that can be used in expressions and conditionals. These can be included in expressions given as strings, as shown in the example below: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL | ||
|
||
# FROM employees | ||
# | KEEP first_name, last_name, height | ||
# | WHERE LENGTH(first_name) < 4" | ||
query = ( | ||
ESQL.from_("employees") | ||
.keep("first_name", "last_name", "height") | ||
.where("LENGTH(first_name) < 4") | ||
) | ||
``` | ||
|
||
All available ES|QL functions have Python wrappers in the `elasticsearch.esql.functions` module, which can be used when building expressions using Python syntax. Below is the example above coded using Python syntax: | ||
|
||
```python | ||
from elasticsearch.esql import ESQL, functions | ||
|
||
# FROM employees | ||
# | KEEP first_name, last_name, height | ||
# | WHERE LENGTH(first_name) < 4" | ||
query = ( | ||
ESQL.from_("employees") | ||
.keep("first_name", "last_name", "height") | ||
.where(functions.length(E("first_name")) < 4) | ||
) | ||
``` | ||
|
||
Note that arguments passed to functions are assumed to be literals. When passing field names, it is necessary to wrap them with the `E()` helper function so that they are interpreted correctly. | ||
|
||
You can find the complete list of available functions in the [ES|QL API reference documentation](https://elasticsearch-py.readthedocs.io/en/stable/esql.html#module-elasticsearch.esql.functions). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
ES|QL Query Builder | ||
=================== | ||
|
||
Commands | ||
-------- | ||
|
||
.. autoclass:: elasticsearch.esql.ESQL | ||
:inherited-members: | ||
:members: | ||
|
||
.. autoclass:: elasticsearch.esql.esql.ESQLBase | ||
:inherited-members: | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.From | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Row | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Show | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.ChangePoint | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Completion | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Dissect | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Drop | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Enrich | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Eval | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Fork | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Grok | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Keep | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Limit | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.LookupJoin | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.MvExpand | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Rename | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Sample | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Sort | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Stats | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
.. autoclass:: elasticsearch.esql.esql.Where | ||
:members: | ||
:exclude-members: __init__ | ||
|
||
Functions | ||
--------- | ||
|
||
.. automodule:: elasticsearch.esql.functions | ||
:members: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what your thoughts are about being able to pass the query object directly here and have the conversion to string in the generated method. Maybe something we can think about for the future as an improvement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good idea, and I don't think this can be a problem in the future