You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Kusto is an ad-hoc query engine that hosts large datasets and
15
-
attempts to satisfy queries by holding all relevant data in-memory.
16
-
There's an inherent risk that queries will monopolize the service
17
-
resources without bounds. Kusto provides several built-in protections
18
-
in the form of default query limits. If you're considering removing these limits, first determine whether you actually gain any value by doing so.
13
+
Kusto is an ad-hoc query engine that hosts large datasets and tries to satisfy queries by holding all relevant data in memory. There's an inherent risk that queries monopolize the service resources without bounds. Kusto provides several built-in protections in the form of default query limits. If you're considering removing these limits, first determine whether you actually gain any value by doing so.
19
14
20
15
## Limit on request concurrency
21
16
22
-
**Request concurrency** is a limit that is imposed on several requests running at the same time.
17
+
**Request concurrency** is a limit on several requests running at the same time.
23
18
24
19
* The default value of the limit depends on the SKU the database is running on, and is calculated as: `Cores-Per-Node x 10`.
25
20
* For example, for a database that's set up on D14v2 SKU, where each machine has 16 vCores, the default limit is `16 cores x10 = 160`.
26
-
*The default value can be changed by configuring the [request rate limit policy](../management/request-rate-limit-policy.md) of the `default` workload group.
27
-
*The actual number of requests that can run concurrently on a database depends on various factors. The most dominant factors are database SKU, database's available resources, and usage patterns. The policy can be configured based on load tests performed on production-like usage patterns.
21
+
*You can change the default value by configuring the [request rate limit policy](../management/request-rate-limit-policy.md) of the `default` workload group.
22
+
*Various factors affect the actual number of requests that can run concurrently on a database. The most dominant factors are database SKU, database's available resources, and usage patterns. Configure the policy based on load tests performed on production-like usage patterns.
28
23
29
24
::: moniker range="azure-data-explorer"
30
25
For more information, see [Optimize for high concurrency with Azure Data Explorer](/azure/data-explorer/high-concurrency).
31
26
::: moniker-end
32
27
33
28
## Limit on result set size (result truncation)
34
29
35
-
**Result truncation** is a limit set by default on the
30
+
**Result truncation** is a default limit on the
36
31
result set returned by the query. Kusto limits the number of records
37
32
returned to the client to **500,000**, and the overall data size for those
38
33
records to **64 MB**. When either of these limits is exceeded, the
39
-
query fails with a "partial query failure". Exceeding overall data size
40
-
will generate an exception with the message:
34
+
query fails with a "partial query failure". Exceeding the overall data size
35
+
generates an exception with the following message:
41
36
42
37
```txt
43
38
The Kusto DataEngine has failed to execute a query: 'Query result set has exceeded the internal data size limit 67108864 (E_QUERY_RESULT_SET_TOO_LARGE).'
44
39
```
45
40
46
-
Exceeding the number of records will fail with an exception that says:
41
+
Exceeding the number of records fails with an exception that says:
47
42
48
43
```txt
49
44
The Kusto DataEngine has failed to execute a query: 'Query result set has exceeded the internal record count limit 500000 (E_QUERY_RESULT_SET_TOO_LARGE).'
50
45
```
51
46
52
-
There are several strategies for dealing with this error.
47
+
You can use several strategies to resolve this error.
53
48
54
49
* Reduce the result set size by modifying the query to only return interesting data. This strategy is useful when the initial failing query is too "wide". For example, the query doesn't project away data columns that aren't needed.
55
-
* Reduce the result set size by shifting post-query processing, such as aggregations, into the query itself. The strategy is useful in scenarios where the output of the query is fed to another processing system, and that then does other aggregations.
50
+
* Reduce the result set size by shifting post-query processing, such as aggregations, into the query itself. This strategy is useful in scenarios where the output of the query is fed to another processing system, and that system then does other aggregations.
56
51
* Switch from queries to using [data export](../management/data-export/index.md) when you want to export large sets of data from the service.
57
-
* Instruct the service to suppress this query limit using `set` statements listed below or flags in [client request properties](../api/netfx/client-request-properties.md).
52
+
* Instruct the service to suppress this query limit by using the `set` statements listed in the following section or flags in [client request properties](../api/netfx/client-request-properties.md).
58
53
59
54
Methods for reducing the result set size produced by the query include:
60
55
61
-
* Use the [summarize operator](../query/summarize-operator.md) group and aggregate over
56
+
* Use the [summarize operator](../query/summarize-operator.md)to group and aggregate over
62
57
similar records in the query output. Potentially sample some columns by using the [take_any aggregation function](../query/take-any-aggregation-function.md).
63
58
* Use a [take operator](../query/take-operator.md) to sample the query output.
64
59
* Use the [substring function](../query/substring-function.md) to trim wide free-text columns.
@@ -74,7 +69,7 @@ set notruncation;
74
69
MyTable | take 1000000
75
70
```
76
71
77
-
It's also possible to have more refined control over result truncation
72
+
You can also have more refined control over result truncation
78
73
by setting the value of `truncationmaxsize` (maximum data size in bytes,
79
74
defaults to 64 MB) and `truncationmaxrecords` (maximum number of records,
80
75
defaults to 500,000). For example, the following query sets result truncation
@@ -90,11 +85,11 @@ Removing the result truncation limit means that you intend to move bulk data out
90
85
91
86
You can remove the result truncation limit either for export purposes by using the `.export` command or for later aggregation. If you choose later aggregation, consider aggregating by using Kusto.
92
87
93
-
Kusto provides a number of client libraries that can handle "infinitely large" results by streaming them to the caller.
88
+
Kusto provides many client libraries that can handle "infinitely large" results by streaming them to the caller.
94
89
Use one of these libraries, and configure it to streaming mode.
95
90
For example, use the .NET Framework client (Microsoft.Azure.Kusto.Data) and either set the streaming property of the connection string to *true*, or use the *ExecuteQueryV2Async()* call that always streams results. For an example of how to use *ExecuteQueryV2Async()*, see the [HelloKustoV2](https://github.com/Azure/azure-kusto-samples-dotnet/tree/master/client/HelloKustoV2) application.
96
91
97
-
You may also find the C# streaming ingestion sample application helpful.
92
+
You might also find the C# streaming ingestion sample application helpful.
98
93
99
94
Result truncation is applied by default, not just to the result stream returned to the client.
100
95
:::moniker range="azure-data-explorer"
@@ -107,62 +102,48 @@ It's also applied by default to any subquery that one Eventhouse issues to anoth
107
102
108
103
### Setting multiple result truncation properties
109
104
110
-
The following apply when using `set` statements, and/or when specifying flags in [client request properties](../api/netfx/client-request-properties.md).
111
-
112
-
* If `notruncation` is set, and any of `truncationmaxsize`, `truncationmaxrecords`, or `query_take_max_records` are also set - `notruncation` is ignored.
113
-
* If `truncationmaxsize`, `truncationmaxrecords` and/or `query_take_max_records` are set multiple times - the *lower* value for each property applies.
114
-
115
-
## Limit on memory consumed by query operators (E_RUNAWAY_QUERY)
105
+
The following rules apply when you use `set` statements or specify flags in [client request properties](../api/netfx/client-request-properties.md).
116
106
117
-
Kusto limits the memory that each query operator can consume to protect against "runaway" queries.
118
-
This limit might be reached by some query operators, such as `join` and `summarize`, that operate by
119
-
holding significant data in memory. By default the limit is 5GB (per node), and it can be increased by setting the request option
120
-
`maxmemoryconsumptionperiterator`:
107
+
* If you set `notruncation` but also set `truncationmaxsize`, `truncationmaxrecords`, or `query_take_max_records`, the service ignores `notruncation`.
108
+
* If you set `truncationmaxsize`, `truncationmaxrecords`, or `query_take_max_records` more than once, the service uses the *lower* value for each property.
121
109
122
-
<!-- csl -->
123
-
```kusto
124
-
set maxmemoryconsumptionperiterator=16106127360;
125
-
MyTable | summarize count() by Use
126
-
```
110
+
## Limit on memory consumed by query operators
127
111
128
-
When this limit is reached, a partial query failure is emitted with a message that includes the text `E_RUNAWAY_QUERY`.
112
+
**Max memory consumption per iterator**limit can be configured to control the amount of memory that each query operator consumes, per node. Some query operators, such as `join` and `summarize`, hold significant data in memory. By increasing the default of the request option `maxmemoryconsumptionperiterator`, you can run queries that require more memory per operator.
129
113
130
-
```text
131
-
The ClusterBy operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete E_RUNAWAY_QUERY.
114
+
The maximum supported value for this request option is 32212254720 (30 GB). If you set `maxmemoryconsumptionperiterator` multiple times, for example in both client request properties and using a `set` statement, the lower value applies.
132
115
133
-
The DemultiplexedResultSetCache operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
116
+
When the query reaches the configured memory per operator limit, a partial query failure message displays and includes the text `E_RUNAWAY_QUERY`.
134
117
135
-
The ExecuteAndCache operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
118
+
For example:
136
119
137
-
The HashJoin operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
120
+
`The ClusterBy operator has exceeded the memory budget during evaluation. Results might be incorrect or incomplete (E_RUNAWAY_QUERY).`
138
121
139
-
The Sort operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
122
+
`The HashJoin operator has exceeded the memory budget during evaluation. Results might be incorrect or incomplete (E_RUNAWAY_QUERY).`
140
123
141
-
The Summarize operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
124
+
`The Sort operator has exceeded the memory budget during evaluation. Results might be incorrect or incomplete (E_RUNAWAY_QUERY).`
142
125
143
-
The TopNestedAggregator operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
126
+
For example, this query sets the max memory consumption per iterator to 15 GB:
144
127
145
-
The TopNested operator has exceeded the memory budget during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY).
128
+
<!-- csl -->
129
+
```kusto
130
+
set maxmemoryconsumptionperiterator=16106127360;
131
+
MyTable | summarize count() by Use
146
132
```
147
133
148
-
If `maxmemoryconsumptionperiterator` is set multiple times, for example in both client request properties and using a `set` statement, the lower value applies.
134
+
Another limit that might trigger an `E_RUNAWAY_QUERY` partial query failure is the max accumulated size of
135
+
strings held by a single operator. The request option above can't override this limit.
149
136
150
-
The maximum supported value for this request option is 32212254720 (30 GB).
151
-
152
-
An additional limit that might trigger an `E_RUNAWAY_QUERY` partial query failure is a limit on the max accumulated size of
153
-
strings held by a single operator. This limit cannot be overridden by the request option above:
154
-
155
-
```text
156
-
Runaway query (E_RUNAWAY_QUERY). Aggregation over string column exceeded the memory budget of 8GB during evaluation.
157
-
```
137
+
`Runaway query (E_RUNAWAY_QUERY). Aggregation over string column exceeded the memory budget of 8GB during evaluation.`
158
138
159
139
When this limit is exceeded, most likely the relevant query operator is a `join`, `summarize`, or `make-series`.
160
-
To work-around the limit, one should modify the query to use the [shuffle query](../query/shuffle-query.md) strategy.
161
-
(This is also likely to improve the performance of the query.)
140
+
141
+
To work around the limit, modify the query to use the [shuffle query](../query/shuffle-query.md) strategy. This change is also likely to improve the performance of the query.
162
142
163
143
In all cases of `E_RUNAWAY_QUERY`, an additional option (beyond increasing the limit by setting the request option and changing the
164
-
query to use a shuffle strategy) is to switch to sampling.
165
-
The two queries below show how to do the sampling. The first query is a statistical sampling, using a random number generator. The second query is deterministic sampling, done by hashing some column from the dataset, usually some ID.
144
+
query to use a shuffle strategy) is to switch to sampling. Sampling reduces the amount of data processed by the query, and therefore reduces the memory pressure on query operators.
145
+
146
+
These two queries show how to do the sampling. The first query is a statistical sampling, using a random number generator. The second query is deterministic sampling, done by hashing some column from the dataset, usually some ID.
166
147
167
148
<!-- csl -->
168
149
```kusto
@@ -171,6 +152,8 @@ T | where rand() < 0.1 | ...
171
152
T | where hash(UserId, 10) == 1 | ...
172
153
```
173
154
155
+
For more information about using mechanisms such as hint.shufflekey for both `summarize` and `join`, see [Best practices for Kusto Query Language queries](../query/best-practices.md).
156
+
174
157
## Limit on memory per node
175
158
176
159
**Max memory per query per node** is another limit used to protect against "runaway" queries. This limit, represented by the request option `max_memory_consumption_per_query_per_node`, sets an upper bound
@@ -205,7 +188,7 @@ management commands. This value can be increased if needed (capped at one hour).
205
188
property. For example, in .NET SDK this is done through a [client request property](../api/rest/request-properties.md),
206
189
by setting a value of type `System.TimeSpan`.
207
190
208
-
**Notes about timeouts**
191
+
### Notes about timeouts
209
192
210
193
* On the client side, the timeout is applied from the request being created
211
194
until the time that the response starts arriving to the client. The time it
@@ -225,7 +208,7 @@ management commands. This value can be increased if needed (capped at one hour).
225
208
226
209
Kusto lets you run queries and use all the available CPU resources that the database has.
227
210
It attempts to do a fair round-robin between queries if more than one is running. This method yields the best performance for query-defined functions.
228
-
At other times, you may want to limit the CPU resources used for a particular
211
+
At other times, you might want to limit the CPU resources used for a particular
229
212
query. If you run a "background job", for example, the system might tolerate higher
230
213
latencies to give concurrent inline queries high priority.
231
214
@@ -234,8 +217,8 @@ The properties are *query_fanout_threads_percent* and *query_fanout_nodes_percen
234
217
Both properties are integers that default to the maximum value (100), but may be reduced for a specific query to some other value.
235
218
236
219
The first, *query_fanout_threads_percent*, controls the fanout factor for thread use.
237
-
When this property is set 100%, all CPUs will be assigned on each node. For example, 16 CPUs deployed on Azure D14 nodes.
238
-
When this property is set to 50%, then half of the CPUs will be used, and so on.
220
+
When this property is set 100%, all CPUs are assigned on each node. For example, 16 CPUs deployed on Azure D14 nodes.
221
+
When this property is set to 50%, then half of the CPUs are used, and so on.
239
222
The numbers are rounded up to a whole CPU, so it's safe to set the property value to 0.
240
223
241
224
The second, *query_fanout_nodes_percent*, controls how many of the query nodes to use per subquery distribution operation.
@@ -246,7 +229,7 @@ If `query_fanout_nodes_percent` or `query_fanout_threads_percent` are set multip
246
229
## Limit on query complexity
247
230
248
231
During query execution, the query text is transformed into a tree of relational operators representing the query.
249
-
If the tree depth exceeds an internal threshold, the query is considered too complex for processing, and will fail with an error code. The failure indicates that the relational operators tree exceeds its limits.
232
+
If the tree depth exceeds an internal threshold, the query is considered too complex for processing, and fails with an error code. The failure indicates that the relational operators tree exceeds its limits.
250
233
251
234
The following examples show common query patterns that can cause the query to exceed this limit and fail:
252
235
@@ -267,7 +250,7 @@ T
267
250
| where Column in ("value1", "value2".... "valueN")
268
251
```
269
252
270
-
* a query which has a union operator that is running too wide schema analysis especially that the default flavor of union is to return "outer" union schema (meaning – that output will include all columns of the underlying table).
253
+
* a query which has a union operator that's running too wide schema analysis especially that the default flavor of union is to return "outer" union schema (meaning – that output includes all columns of the underlying table).
271
254
272
255
The suggestion in this case is to review the query and reduce the columns being used by the query.
![learn-build-service-prod[bot]](https://avatars.githubusercontent.com/in/237442?v=4&size=40){kind=avatar}
0 commit comments