You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Document new hypertable API (#2732)
Just make minimal changes to document the new API. Sections on using
the new API will be following.
Co-authored-by: Erik Nordström <[email protected]>
* Fix broken links in distributed hypertable docs (#2767)
* Fix links to dimension builders
* Update api/add_dimension.md
Co-authored-by: James Guthrie <[email protected]>
Signed-off-by: Mats Kindahl <[email protected]>
---------
Signed-off-by: Mats Kindahl <[email protected]>
Co-authored-by: Erik Nordström <[email protected]>
Co-authored-by: James Guthrie <[email protected]>
Copy file name to clipboardExpand all lines: api/add_dimension.md
+30-50
Original file line number
Diff line number
Diff line change
@@ -13,38 +13,40 @@ api:
13
13
14
14
Add an additional partitioning dimension to a Timescale hypertable.
15
15
The column selected as the dimension can either use interval
16
-
partitioning (for example, for a second time partition) or hash partitioning.
16
+
partitioning (for example, for a second range partition) or hash partitioning.
17
+
18
+
**Note: this reference describes the new generalized hypertable API. The [old interface for `add_dimension` is also available](add_dimension_old.md).**
17
19
18
20
<Highlighttype="warning">
19
21
The `add_dimension` command can only be executed after a table has been
20
22
converted to a hypertable (via `create_hypertable`), but must similarly
21
23
be run only on an empty hypertable.
22
24
</Highlight>
23
25
24
-
**Space partitions**: Using space partitions is highly recommended
26
+
**Hash partitions (previously called space partitions)**: Using hash partitions is highly recommended
25
27
for [distributed hypertables][distributed-hypertables] to achieve
26
28
efficient scale-out performance. For [regular hypertables][regular-hypertables]
27
29
that exist only on a single node, additional partitioning can be used
28
30
for specialized use cases and not recommended for most users.
29
31
30
-
Space partitions use hashing: Every distinct item is hashed to one of
31
-
*N* buckets. Remember that we are already using (flexible) time
32
-
intervals to manage chunk sizes; the main purpose of space
32
+
Every distinct item in hash partitioning is hashed to one of
33
+
*N* buckets. Remember that we are already using (flexible) range
34
+
intervals to manage chunk sizes; the main purpose of hash
33
35
partitioning is to enable parallelization across multiple
34
36
data nodes (in the case of distributed hypertables) or
35
37
across multiple disks within the same time interval
36
38
(in the case of single-node deployments).
37
39
38
40
### Parallelizing queries across multiple data nodes
39
41
40
-
In a distributed hypertable, space partitioning enables inserts to be
42
+
In a distributed hypertable, hash partitioning enables inserts to be
41
43
parallelized across data nodes, even while the inserted rows share
42
44
timestamps from the same time interval, and thus increases the ingest rate.
43
45
Query performance also benefits by being able to parallelize queries
44
46
across nodes, particularly when full or partial aggregations can be
45
47
"pushed down" to data nodes (for example, as in the query
46
48
`avg(temperature) FROM conditions GROUP BY hour, location`
47
-
when using `location` as a space partition). Please see our
49
+
when using `location` as a hash partition). Please see our
48
50
[best practices about partitioning in distributed hypertables][distributed-hypertable-partitioning-best-practices]
49
51
for more information.
50
52
@@ -71,10 +73,10 @@ disks, single query to multiple disks in parallel). The multiple
71
73
tablespace approach only supports the former. With a RAID setup,
72
74
*no spatial partitioning is required*.
73
75
74
-
That said, when using space partitions, we recommend using 1
75
-
space partition per disk.
76
+
That said, when using hash partitions, we recommend using 1
77
+
hash partition per disk.
76
78
77
-
Timescale does *not* benefit from a very large number of space
79
+
Timescale does *not* benefit from a very large number of hash
78
80
partitions (such as the number of unique items you expect in partition
79
81
field). A very large number of such partitions leads both to poorer
80
82
per-partition load balancing (the mapping of items to partitions using
@@ -86,7 +88,7 @@ queries.
86
88
|Name|Type|Description|
87
89
|-|-|-|
88
90
|`hypertable`|REGCLASS|Hypertable to add the dimension to|
89
-
|`column_name`|TEXT|Column to partition by|
91
+
|`dimension`|DIMENSION_INFO | Dimension to partition by|
90
92
91
93
### Optional arguments
92
94
@@ -102,66 +104,44 @@ queries.
102
104
|Column|Type|Description|
103
105
|-|-|-|
104
106
|`dimension_id`|INTEGER|ID of the dimension in the TimescaleDB internal catalog|
105
-
|`schema_name`|TEXT|Schema name of the hypertable|
106
-
|`table_name`|TEXT|Table name of the hypertable|
107
-
|`column_name`|TEXT|Column name of the column to partition by|
108
107
|`created`|BOOLEAN|True if the dimension was added, false when `if_not_exists` is true and no dimension was added|
109
108
110
-
When executing this function, either `number_partitions` or
111
-
`chunk_time_interval` must be supplied, which dictates if the
112
-
dimension uses hash or interval partitioning.
113
-
114
-
The `chunk_time_interval` should be specified as follows:
115
-
116
-
* If the column to be partitioned is a TIMESTAMP, TIMESTAMPTZ, or
117
-
DATE, this length should be specified either as an INTERVAL type or
118
-
an integer value in *microseconds*.
119
-
120
-
* If the column is some other integer type, this length
121
-
should be an integer that reflects
122
-
the column's underlying semantics (for example, the
123
-
`chunk_time_interval` should be given in milliseconds if this column
124
-
is the number of milliseconds since the UNIX epoch).
125
-
126
-
<Highlighttype="warning">
127
-
Supporting more than **one** additional dimension is currently
128
-
experimental. For any production environments, users are recommended
129
-
to use at most one "space" dimension.
130
-
131
-
</Highlight>
132
-
133
109
### Sample use
134
110
135
-
First convert table `conditions` to hypertable with just time
136
-
partitioning on column `time`, then add an additional partition key on `location` with four partitions:
111
+
First convert table `conditions` to hypertable with just range
112
+
partitioning on column `time`, then add an additional partition key on
excerpt: Add a space-partitioning dimension to a hypertable
4
+
topics: [hypertables]
5
+
keywords: [hypertables, partitions]
6
+
tags: [dimensions, chunks]
7
+
api:
8
+
license: apache
9
+
type: function
10
+
---
11
+
12
+
# add_dimension()
13
+
14
+
Add an additional partitioning dimension to a Timescale hypertable.
15
+
The column selected as the dimension can either use interval
16
+
partitioning (for example, for a second time partition) or hash partitioning.
17
+
18
+
<Highlighttype="warning">
19
+
The `add_dimension` command can only be executed after a table has been
20
+
converted to a hypertable (via `create_hypertable`), but must similarly
21
+
be run only on an empty hypertable.
22
+
</Highlight>
23
+
24
+
**Space partitions**: Using space partitions is highly recommended
25
+
for [distributed hypertables][distributed-hypertables] to achieve
26
+
efficient scale-out performance. For [regular hypertables][regular-hypertables]
27
+
that exist only on a single node, additional partitioning can be used
28
+
for specialized use cases and not recommended for most users.
29
+
30
+
Space partitions use hashing: Every distinct item is hashed to one of
31
+
*N* buckets. Remember that we are already using (flexible) time
32
+
intervals to manage chunk sizes; the main purpose of space
33
+
partitioning is to enable parallelization across multiple
34
+
data nodes (in the case of distributed hypertables) or
35
+
across multiple disks within the same time interval
36
+
(in the case of single-node deployments).
37
+
38
+
### Parallelizing queries across multiple data nodes
39
+
40
+
In a distributed hypertable, space partitioning enables inserts to be
41
+
parallelized across data nodes, even while the inserted rows share
42
+
timestamps from the same time interval, and thus increases the ingest rate.
43
+
Query performance also benefits by being able to parallelize queries
44
+
across nodes, particularly when full or partial aggregations can be
45
+
"pushed down" to data nodes (for example, as in the query
46
+
`avg(temperature) FROM conditions GROUP BY hour, location`
47
+
when using `location` as a space partition). Please see our
48
+
[best practices about partitioning in distributed hypertables][distributed-hypertable-partitioning-best-practices]
49
+
for more information.
50
+
51
+
### Parallelizing disk I/O on a single node
52
+
53
+
Parallel I/O can benefit in two scenarios: (a) two or more concurrent
54
+
queries should be able to read from different disks in parallel, or
55
+
(b) a single query should be able to use query parallelization to read
56
+
from multiple disks in parallel.
57
+
58
+
Thus, users looking for parallel I/O have two options:
59
+
60
+
1. Use a RAID setup across multiple physical disks, and expose a
61
+
single logical disk to the hypertable (that is, via a single tablespace).
62
+
63
+
1. For each physical disk, add a separate tablespace to the
64
+
database. Timescale allows you to actually add multiple tablespaces
65
+
to a *single* hypertable (although under the covers, a hypertable's
66
+
chunks are spread across the tablespaces associated with that hypertable).
67
+
68
+
We recommend a RAID setup when possible, as it supports both forms of
69
+
parallelization described above (that is, separate queries to separate
70
+
disks, single query to multiple disks in parallel). The multiple
71
+
tablespace approach only supports the former. With a RAID setup,
72
+
*no spatial partitioning is required*.
73
+
74
+
That said, when using space partitions, we recommend using 1
75
+
space partition per disk.
76
+
77
+
Timescale does *not* benefit from a very large number of space
78
+
partitions (such as the number of unique items you expect in partition
79
+
field). A very large number of such partitions leads both to poorer
80
+
per-partition load balancing (the mapping of items to partitions using
81
+
hashing), as well as much increased planning latency for some types of
82
+
queries.
83
+
84
+
### Required arguments
85
+
86
+
|Name|Type|Description|
87
+
|-|-|-|
88
+
|`hypertable`|REGCLASS|Hypertable to add the dimension to|
89
+
|`column_name`|TEXT|Column to partition by|
90
+
91
+
### Optional arguments
92
+
93
+
|Name|Type|Description|
94
+
|-|-|-|
95
+
|`number_partitions`|INTEGER|Number of hash partitions to use on `column_name`. Must be > 0|
96
+
|`chunk_time_interval`|INTERVAL|Interval that each chunk covers. Must be > 0|
97
+
|`partitioning_func`|REGCLASS|The function to use for calculating a value's partition (see `create_hypertable`[instructions][create_hypertable])|
98
+
|`if_not_exists`|BOOLEAN|Set to true to avoid throwing an error if a dimension for the column already exists. A notice is issued instead. Defaults to false|
99
+
100
+
### Returns
101
+
102
+
|Column|Type|Description|
103
+
|-|-|-|
104
+
|`dimension_id`|INTEGER|ID of the dimension in the TimescaleDB internal catalog|
105
+
|`schema_name`|TEXT|Schema name of the hypertable|
106
+
|`table_name`|TEXT|Table name of the hypertable|
107
+
|`column_name`|TEXT|Column name of the column to partition by|
108
+
|`created`|BOOLEAN|True if the dimension was added, false when `if_not_exists` is true and no dimension was added|
109
+
110
+
When executing this function, either `number_partitions` or
111
+
`chunk_time_interval` must be supplied, which dictates if the
112
+
dimension uses hash or interval partitioning.
113
+
114
+
The `chunk_time_interval` should be specified as follows:
115
+
116
+
* If the column to be partitioned is a TIMESTAMP, TIMESTAMPTZ, or
117
+
DATE, this length should be specified either as an INTERVAL type or
118
+
an integer value in *microseconds*.
119
+
120
+
* If the column is some other integer type, this length
121
+
should be an integer that reflects
122
+
the column's underlying semantics (for example, the
123
+
`chunk_time_interval` should be given in milliseconds if this column
124
+
is the number of milliseconds since the UNIX epoch).
125
+
126
+
<Highlighttype="warning">
127
+
Supporting more than **one** additional dimension is currently
128
+
experimental. For any production environments, users are recommended
129
+
to use at most one "space" dimension.
130
+
131
+
</Highlight>
132
+
133
+
### Sample use
134
+
135
+
First convert table `conditions` to hypertable with just time
136
+
partitioning on column `time`, then add an additional partition key on `location` with four partitions:
0 commit comments