Skip to content

Commit efb7776

Browse files
rafaelasantanaamotl
rafaelasantana
authored and
amotl
committed
Prometheus: Starter tutorial
1 parent 427d701 commit efb7776

File tree

3 files changed

+162
-2
lines changed

3 files changed

+162
-2
lines changed

‎docs/conf.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,8 @@
7171
# 429 Client Error: Too Many Requests
7272
r"https://www.terraform.io",
7373
r"https://developer.hashicorp.com",
74+
# 403 Client Error: Forbidden for url
75+
r"https://www.computerhope.com/",
7476
]
7577

7678
linkcheck_anchors_ignore_for_url += [

‎docs/integrate/prometheus/index.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,8 @@ Getting started with Prometheus and CrateDB for long-term storage.
6464
:::
6565

6666
:::{grid-item-card} Tutorial: Storing long-term metrics with Prometheus in CrateDB
67-
:link: https://community.cratedb.com/t/storing-long-term-metrics-with-prometheus-in-cratedb/1012
68-
:link-type: url
67+
:link: prometheus-tutorial
68+
:link-type: ref
6969
Set up CrateDB as a long-term metrics store for Prometheus using Docker Compose.
7070
:::
7171

@@ -112,6 +112,13 @@ tutorial.
112112
```
113113

114114

115+
:::{toctree}
116+
:maxdepth: 1
117+
:hidden:
118+
Tutorial <tutorial>
119+
:::
120+
121+
115122
[CrateDB]: https://github.com/crate/crate
116123
[CrateDB and Prometheus]: https://cratedb.com/integrations/cratedb-and-prometheus
117124
[CrateDB Prometheus Adapter]: https://github.com/crate/cratedb-prometheus-adapter

‎docs/integrate/prometheus/tutorial.md‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
(prometheus-tutorial)=
2+
# Storing long-term metrics with Prometheus in CrateDB
3+
4+
In this tutorial, I show how to
5+
6+
* Set up Docker Compose to run CrateDB, Prometheus, and the CrateDB Prometheus Adapter
7+
* Run the applications with Docker Compose
8+
9+
*Note: this blog post uses CrateDB 4.7.0, Prometheus 2.33.3 and CrateDB Prometheus Adapter 0.4.0*
10+
11+
## Motivation
12+
13+
[Prometheus](https://prometheus.io/docs/introduction/overview/) is a monitoring software for collecting metrics data from applications and infrastructures. Its focus lies on collecting big amounts of concise event data from the monitored system, roughly timestamp points with key-value pairs.
14+
15+
Such data is very useful to track the state and trajectory of a system, so storing this data for the long term is a common need for Prometheus users.
16+
17+
This is where [CrateDB](https://cratedb.com/database) comes into place. With the [CrateDB Prometheus Adapter](https://github.com/crate/cratedb-prometheus-adapter), one can easily store the collected metrics data in CrateDB and take advantage of its high ingestion and query speed and friendly UI to massively scale-out Prometheus.
18+
19+
## Set up Docker Compose
20+
21+
Both CrateDB, Prometheus, and the CrateDB Prometheus Adapter applications can be run as [Docker containers](https://www.docker.com/resources/what-container). To then centralize the container management I use [Docker Compose](https://docs.docker.com/compose/), this way I can build and run all the containers with a single command and set up the connections between them in a YAML file.
22+
23+
Before anything else, I follow the [Docker Installation Tutorial](https://docs.docker.com/get-docker/) to get Docker in my local machine.
24+
25+
Then, I create a directory in my local machine to host the necessary configuration files.
26+
I’ll have a total of three of them, all following the YAML format. They can be easily created using any [text editor](https://www.computerhope.com/jargon/e/editor.htm), like TextEdit on a Mac, and then saved with the `.yml` format.
27+
28+
### Create `docker-compose.yml`
29+
30+
The first YAML file I create is `docker-compose.yml`, which wraps up the configurations for the three containers.
31+
32+
I specify CrateDB, Prometheus, and Adapter as services. Then, I add `config.yml` and `prometheus.yml` files as volumes to the Adapter and Prometheus containers, respectively. These files will be created in the following steps.
33+
```yaml
34+
services:
35+
cratedb:
36+
image: "crate"
37+
ports:
38+
- "4200:4200"
39+
- "5432:5432"
40+
prometheus:
41+
image: "prom/prometheus"
42+
volumes:
43+
- ./prometheus.yml:/etc/prometheus/prometheus.yml
44+
ports:
45+
- "9090:9090"
46+
cratedb-prometheus-adapter:
47+
image: "ghcr.io/crate/cratedb-prometheus-adapter"
48+
volumes:
49+
- ./config.yml:/etc/cratedb-prometheus-adapter/config.yml
50+
ports:
51+
- "9268:9268"
52+
```
53+
54+
### Create `prometheus.yml`
55+
56+
Next, following the[ Prometheus Documentation](https://prometheus.io/docs/prometheus/latest/getting_started/), I create a `prometheus.yml` file, which holds the scraping configuration for whichever service Prometheus collects metrics from.
57+
58+
To keep it simple, I follow the example in the Prometheus documentation and set it to monitor itself.
59+
60+
One last bit of configuration necessary to forward requests from Prometheus to the CrateDB Adapter is to set `remote_write` and `remote_read` to the Adapter URL, as stated in [CrateDB Prometheus Adapter Setup](https://github.com/crate/cratedb-prometheus-adapter).
61+
62+
As I’m running the Adapter on Docker instead of locally, the host in its URL will not be `localhost`, but rather however I called the Adapter service previously in my `docker-compose.yml` file, in this case, `cratedb-prometheus-adapter`.
63+
64+
The resulting prometheus.yml looks then like this:
65+
```yaml
66+
global:
67+
scrape_interval: 15s # By default, scrape targets every 15 seconds.
68+
69+
# Attach these labels to any time series or alerts when communicating with
70+
# external systems (federation, remote storage, Alertmanager).
71+
external_labels:
72+
monitor: 'codelab-monitor'
73+
74+
# A scrape configuration containing exactly one endpoint to scrape:
75+
# Here it's Prometheus itself.
76+
scrape_configs:
77+
# The job name is added as a label `job=<job_name>` to any time-series scraped from this config.
78+
- job_name: 'prometheus'
79+
80+
# Override the global default and scrape targets from this job every 5 seconds.
81+
scrape_interval: 5s
82+
83+
static_configs:
84+
- targets: ['localhost:9090']
85+
86+
remote_write:
87+
- url: http://cratedb-prometheus-adapter:9268/write
88+
remote_read:
89+
- url: http://cratedb-prometheus-adapter:9268/read
90+
```
91+
### Create `config.yml`
92+
93+
Finally, following the [CrateDB Prometheus Adapter setup instructions](https://github.com/crate/cratedb-prometheus-adapter), I create the `config.yml` file, which defines the CrateDB endpoints the Adapter writes to.
94+
95+
As I did previously in the `prometheus.yml` file, the host is set to `cratedb`, which is how I declared the CrateDB service on the `docker-compose.yml` file, instead of the default `localhost`. The remaining variables are set with their default values.
96+
```yaml
97+
cratedb_endpoints:
98+
- host: "cratedb" # Host to connect to (default: "localhost")
99+
port: 5432 # Port to connect to (default: 5432).
100+
user: "crate" # Username to use (default: "crate")
101+
password: "" # Password to use (default: "").
102+
schema: "" # Schema to use (default: "").
103+
connect_timeout: 10 # TCP connect timeout (seconds) (default: 10).
104+
max_connections: 5 # The maximum number of concurrent connections (default: 5).
105+
enable_tls: false # Whether to connect using TLS (default: false).
106+
allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).
107+
```
108+
I make sure both `docker-compose.yml`, `config.yml`, and `prometheus.yml` are in the same directory in my local machine.
109+
110+
## Start the services
111+
112+
Finally, I navigate to my CrateDB-Prometheus directory in my terminal and start Docker Compose with the `docker-compose up` command
113+
```shell
114+
$ cd /Users/Path/To/Directory/CrateDB-Prometheus
115+
$ docker-compose up
116+
```
117+
118+
With Docker Compose up and running, I follow the [CrateDB Prometheus Adapter setup instructions](https://github.com/crate/cratedb-prometheus-adapter), navigate to the [CrateDB Admin UI](https://www.google.com/search?client=safari&rls=en&q=cratedb+admin+ui&ie=UTF-8&oe=UTF-8) at [http://localhost:4200](http://localhost:4200/) and create a `metrics` table in CrateDB, which will store the metrics gathered by Prometheus.
119+
120+
```sql
121+
CREATE TABLE "metrics" (
122+
"timestamp" TIMESTAMP,
123+
"labels_hash" STRING,
124+
"labels" OBJECT(DYNAMIC),
125+
"value" DOUBLE,
126+
"valueRaw" LONG,
127+
"day__generated" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
128+
PRIMARY KEY ("timestamp", "labels_hash", "day__generated")
129+
) PARTITIONED BY ("day__generated")
130+
```
131+
Then I navigate to [http://localhost:9090](http://localhost:9090/), where I find the Prometheus UI. There, I head to **Status** and then **Targets**
132+
133+
![im1|690x206](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/91223397b30bce2f7188617436ea12ceed83d83c.png)
134+
135+
And confirm that Prometheus is successfully monitoring itself.
136+
137+
![im2|690x173](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/57ccb5374b0ab524466de08feefbafde559dac87.png)
138+
139+
Lastly, I head back to the CrateDB Admin UI and select the `metrics` table I just created.
140+
141+
I see that only after a few minutes of running, Prometheus has gathered over 300k data points.
142+
143+
![im3|690x403](https://us1.discourse-cdn.com/flex020/uploads/crate/original/1X/22e8c7d5a90ec9240a4cb4269774e143759aa92e.jpeg)
144+
145+
I can now enjoy CrateDB’s incredible query speed to analyze and visualize this
146+
data using tools like {ref}`grafana`, which works effortlessly with CrateDB.
147+
148+
Here are a few interesting tutorials on that matter:
149+
150+
* https://cratedb.com/blog/visualizing-time-series-data-with-grafana-and-cratedb
151+
* https://cratedb.com/blog/monitoring-cratedb-with-prometheus-and-grafana

0 commit comments

Comments
 (0)