Skip to content

Commit 648a82c

Browse files
learn-build-service-prod[bot]learn-build-service-prod[bot]
authored
Merge pull request #2818 from MicrosoftDocs/main639034057742314886sync_temp
For protected branch, push strategy should use PR and merge to target branch method to work around git push error
2 parents eaae73a + 17abf6a commit 648a82c

6 files changed

Lines changed: 166 additions & 36 deletions

File tree

‎data-explorer/kusto/functions-library/slm-embeddings-fl.md‎

Lines changed: 18 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -135,25 +135,26 @@ let slm_embeddings_fl=(tbl:(*), text_col:string, embeddings_col:string, batch_si
135135
{
136136
let kwargs = bag_pack('text_col', text_col, 'embeddings_col', embeddings_col, 'batch_size', batch_size, 'model_name', model_name, 'prefix', prefix);
137137
let code = ```if 1:
138-
from sandbox_utils import Zipackage
139-
Zipackage.install('embedding_engine.zip')
140-
141-
from embedding_factory import create_embedding_engine
142-
143-
text_col = kargs["text_col"]
144-
embeddings_col = kargs["embeddings_col"]
145-
batch_size = kargs["batch_size"]
146-
model_name = kargs["model_name"]
147-
prefix = kargs["prefix"]
148-
149-
Zipackage.install(f'{model_name}.zip')
138+
from sandbox_utils import Zipackage
139+
Zipackage.install('embedding_engine.zip')
140+
# Zipackage.install('tokenizers-0.22.1.whl') # redundant if tokenizers package is included in the Python image
141+
142+
from embedding_factory import create_embedding_engine
143+
144+
text_col = kargs["text_col"]
145+
embeddings_col = kargs["embeddings_col"]
146+
batch_size = kargs["batch_size"]
147+
model_name = kargs["model_name"]
148+
prefix = kargs["prefix"]
150149
151-
engine = create_embedding_engine(model_name, cache_dir="C:\\Temp")
152-
embeddings = engine.encode(df[text_col].tolist(), batch_size=batch_size, prefix=prefix)
150+
Zipackage.install(f'{model_name}.zip')
153151
154-
result = df
155-
result[embeddings_col] = list(embeddings)
156-
```;
152+
engine = create_embedding_engine(model_name, cache_dir="C:\\Temp")
153+
embeddings = engine.encode(df[text_col].tolist(), batch_size=batch_size, prefix=prefix) # prefix is used only for E5
154+
155+
result = df
156+
result[embeddings_col] = list(embeddings)
157+
```;
157158
tbl
158159
| evaluate hint.distribution=per_node python(typeof(*), code, kwargs, external_artifacts = bag_pack(
159160
'embedding_engine.zip', 'https://artifactswestus.z22.web.core.windows.net/models/SLM/embedding_engine.zip',

‎data-explorer/kusto/management/alter-merge-table-update-policy-command.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@ Changes the table's update policy. The [update policy](update-policy.md) simplif
1515
>
1616
> * The source table and the table for which the update policy is defined must be in the same database.
1717
> * The update policy function schema and the target table schema must match in their column names, types, and order.
18-
> * If the policy already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy.
1918
2019
## Permissions
2120

@@ -31,14 +30,17 @@ You must have at least [Table Admin](../access-control/role-based-access-control
3130

3231
|Name|Type|Required|Description|
3332
|--|--|--|--|
34-
| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* is not required. |
33+
| *DatabaseName* | `string` | | The name of the database. When you run the command from the database context that contains the table to alter, *DatabaseName* isn't required. |
3534
| *TableName* | `string` | :heavy_check_mark: | The name of the table. A wildcard, `*`, denotes all tables.|
3635
| *ArrayOfPolicyObjects* | `string` | :heavy_check_mark: | A serialized array of policy objects. For more information, see [update policy](update-policy.md).|
3736

3837
## Returns
3938

4039
Returns a JSON representation of the policy.
4140

41+
> [!WARNING]
42+
> If an update policy is already defined on the table, the PolicyObjects specified in the command are added to the array of PolicyObjects in the existing policy. Entries aren't deduplicated, so if the exact same PolicyObject already exists in the current update policy, another entry will be added, potentially causing duplicates during ingestion time.
43+
4244
## Example
4345

4446
The following command changes the update policy for a table using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals).

‎data-explorer/kusto/management/alter-table-update-policy-command.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,9 @@ ms.date: 08/11/2024
1212
Use this command to change the table update policy. The [update policy](update-policy.md) simplifies the process of syncing and updating data between two tables. When new data is inserted into the source table, a transformation query runs over this data to modify and insert the data into the target table.
1313

1414
> [!NOTE]
15-
> The source table and the table for which the update policy is defined must be in the same database.
16-
> The update policy function schema and the target table schema must match in their column names, types, and order.
15+
>
16+
> * The source table and the table for which the update policy is defined must be in the same database.
17+
> * The update policy function schema and the target table schema must match in their column names, types, and order.
1718
1819
## Permissions
1920

@@ -37,6 +38,11 @@ You must have at least [Table Admin](../access-control/role-based-access-control
3738

3839
Returns a JSON representation of the policy.
3940

41+
> [!WARNING]
42+
> The alter command *replaces* the current update policy with a new policy. If you would like to *add* a new entry to the array
43+
> of policy objects, in addition to the current ones, use the
44+
> [.alter-merge table policy update command](alter-merge-table-update-policy-command.md).
45+
4046
## Example
4147

4248
Change the update policy for a table (using [multi-line string literals](../query/scalar-data-types/string.md#multi-line-string-literals)):

‎data-explorer/kusto/management/materialized-views/materialized-view-create.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ There are two possible ways to create a materialized view, as noted by the *back
2222

2323
* See [Backfill a materialized view](#backfill-a-materialized-view).
2424
* Creation might take a long while to complete, depending on the number of records in the source table. The view isn't available for queries until backfill is complete.
25-
* When you're using this option, the create command must be `async`. You can monitor execution by using the [`.show operations`](../show-operations.md) command.
25+
* When you're using this option, the create command must be `async`. You can monitor execution by using the [`.show operations`](../show-operations.md) command. Query for the entire operation log to view the progress of the backfill (see [examples here](../show-operations.md#example)).
2626
* You can cancel the backfill process by using the [`.cancel operation`](#cancel-materialized-view-creation) command.
2727

2828
> [!IMPORTANT]

‎data-explorer/kusto/management/show-operations.md‎

Lines changed: 43 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,7 @@ Entries are appended to the log when operations start and change their state, in
1515
Users can view the ongoing and past operations they started by using the `.show operations` command.
1616
Database administrators can view all operations that apply to the databases they administer.
1717

18-
Users can also view the results of an operation by using the [`.show operation details`](show-operation-details.md)
19-
command. Normally, the results are returned as part of `.show operations` command itself. For asynchronous
20-
management commands, the `.show operation details` command is the only way to view the command's results.
18+
The `.show operations` command returns general details about all operations running on the cluster. Some of the operations also support retrieving the operation's results by using the [`.show operation details`](show-operation-details.md) command.
2119

2220
## Syntax
2321

@@ -33,7 +31,8 @@ management commands, the `.show operation details` command is the only way to vi
3331

3432
## Returns
3533

36-
When an operation ID is omitted, the command returns a table displaying all administrative operations executed in the last two weeks, whether ongoing or completed. It includes entries accessible to the user, with multiple records possible for a single operation. Only one record indicates the terminal state of 'Completed' or 'Failed.' This mode is for checking the history of operations, but note that records may take time to appear in the historic log.
34+
When an operation ID is omitted, the command returns a table displaying all administrative operations executed in the last two weeks, whether ongoing or completed. It includes entries accessible to the user, with multiple records possible for a single operation. Only one record indicates the terminal state of 'Completed' or 'Failed.' This mode is for checking the history of operations. Records may take a short time to appear in the historical log.
35+
You can use [arg_max()](../query/arg-max-aggregation-function.md) over the results of the historical log to view the latest state for each operation ID (see [examples](#example)).
3736

3837
When one or more operation IDs are provided, the command returns the latest update for each ID, given the user's access and the record being less than 6 hours old. This mode helps quickly check the latest status of recently executed operations.
3938

@@ -69,17 +68,49 @@ The following table describes the possible values for the result table's *State*
6968

7069
## Example
7170

71+
The following command returns the latest state per operation ID for operations that started after `2026-01-05`:
72+
73+
```kusto
74+
.show operations
75+
| where StartedOn > datetime(2026-01-05)
76+
| summarize arg_max(LastUpdatedOn, *) by OperationId
77+
| project OperationId, Operation, StartedOn, LastUpdatedOn, Duration, State
78+
```
79+
80+
|OperationId|Operation|StartedOn|LastUpdatedOn|Duration|State|
81+
|---|---|---|---|---|---|
82+
|62e40fad-516c-4133-814f-f509e889d006|DataIngestPull|2026-01-05 18:57:10.4234023|2026-01-05 18:57:53.4074572|00:00:42.9840549|Completed|
83+
|989b527f-20da-48fe-ae22-deba91e20764|OperationsShow|2026-01-05 18:57:54.1959438|2026-01-05 18:57:54.2036051|00:00:00.0076613|Completed|
84+
|1671b635-b42c-45c6-928c-ad3f3436cb75|TableAppend|2026-01-05 18:58:13.3479575|2026-01-05 18:58:13.3974038|00:00:00.0494463|InProgress|
85+
|55bc427e-d576-40dc-bd38-58f9df34d357|DatabasesShow|2026-01-05 18:58:49.4693980|2026-01-05 18:58:49.4697805|00:00:00.0003825|Completed|
86+
87+
The following command returns the entire log (not only latest state) for operation with ID `b152f9da-616a-40a7-8cde-f2390cfc8064`:
88+
89+
```kusto
90+
.show operations
91+
| where LastUpdatedOn > ago(1h)
92+
| where OperationId == "b152f9da-616a-40a7-8cde-f2390cfc8064"
93+
| project OperationId, Operation, StartedOn, LastUpdatedOn, Duration, State, Status
94+
```
95+
96+
|OperationId|Operation|StartedOn|LastUpdatedOn|Duration|State|Status|
97+
|---|---|---|---|---|---|---|
98+
|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:36.8136477|00:00:00.0000758|InProgress||
99+
|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:36.8143127|00:00:00.0006764|InProgress|Assigned|
100+
|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:36.8146015|00:00:00.0009574|InProgress|Assigned|
101+
|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:40.2102817|00:00:03.3966431|InProgress|Extent(s) created; metadata updated; cluster map updated|
102+
|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:40.2103713|00:00:03.3967237|Completed|Extent(s) created; metadata updated; cluster map updated|
103+
104+
The following command returns only the latest state for the same operation. Operations that completed over 6 hours ago will not be returned using this method. Use the options above to query for entries that are older than 6 hours.
105+
72106
```kusto
73-
.show operations
107+
.show operations b152f9da-616a-40a7-8cde-f2390cfc8064
108+
| project OperationId, Operation, StartedOn, LastUpdatedOn, Duration, State, Status
74109
```
75110

76-
|ID |Operation |Node ID |Started On |Last Updated On |Duration |State |Status |
77-
|--|--|--|--|--|--|--|--|
78-
|3827def6-0773-4f2a-859e-c02cf395deaf |SchemaShow | |2015-01-06 08:47:01.0000000 |2015-01-06 08:47:01.0000000 |0001-01-01 00:00:00.0000000 |Completed |
79-
|841fafa4-076a-4cba-9300-4836da0d9c75 |DataIngestPull |Kusto.Azure.Svc_IN_1 |2015-01-06 08:47:02.0000000 |2015-01-06 08:48:19.0000000 |0001-01-01 00:01:17.0000000 |Completed |
80-
|e198c519-5263-4629-a158-8d68f7a1022f |OperationsShow | |2015-01-06 08:47:18.0000000 |2015-01-06 08:47:18.0000000 |0001-01-01 00:00:00.0000000 |Completed |
81-
|a9f287a1-f3e6-4154-ad18-b86438da0929 |ExtentsDrop | |2015-01-11 08:41:01.0000000 |0001-01-01 00:00:00.0000000 |0001-01-01 00:00:00.0000000 |InProgress |
82-
|9edb3ecc-f4b4-4738-87e1-648eed2bd998 |DataIngestPull | |2015-01-10 14:57:41.0000000 |2015-01-10 14:57:41.0000000 |0001-01-01 00:00:00.0000000 |Failed |Collection was modified. Enumeration operation may not execute. |
111+
|OperationId|Operation|StartedOn|LastUpdatedOn|Duration|State|Status|
112+
|---|---|---|---|---|---|---|
113+
|b152f9da-616a-40a7-8cde-f2390cfc8064|DataIngestPull|2026-01-06 09:33:36.8136476|2026-01-06 09:33:40.2103713|00:00:03.3967237|Completed|Extent(s) created; metadata updated; cluster map updated|
83114

84115
## Related content
85116

‎data-explorer/kusto/management/update-policy.md‎

Lines changed: 92 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,8 +92,9 @@ Each such object is represented as a JSON property bag, with the following prope
9292
|Property |Type | Description |
9393
|---------|---------|----------------|
9494
|IsEnabled |`bool` |States if update policy is *true* - enabled, or *false* - disabled|
95-
|Source |`string` |Name of the table that triggers invocation of the update policy |
96-
|Query |`string` |A query used to produce data for the update |
95+
|Source |`string` |Name of the table that triggers invocation of the update policy. |
96+
|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. See [Update policy with source table wildcard pattern](#update-policy-with-source-table-wildcard-pattern) |
97+
|Query |`string` |A query used to produce data for the update. |
9798
|IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. |
9899
|PropagateIngestionProperties |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. |
99100
|ManagedIdentity | `string` | The managed identity on behalf of which the update policy runs. The managed identity can be an object ID, or the `system` reserved word. The update policy must be configured with a managed identity when the query references tables in other databases or tables with an enabled [row level security policy](row-level-security-policy.md). For more information, see [Use a managed identity to run a update policy](update-policy-with-managed-identity.md). |
@@ -105,6 +106,7 @@ Each such object is represented as a JSON property bag, with the following prope
105106
|---------|---------|----------------|
106107
|IsEnabled |`bool` |States if update policy is *true* - enabled, or *false* - disabled|
107108
|Source |`string` |Name of the table that triggers invocation of the update policy |
109+
|SourceIsWildCard |`bool` |If *true*, the `Source` property can be a wildcard pattern. |
108110
|Query |`string` |A query used to produce data for the update |
109111
|IsTransactional |`bool` |States if the update policy is transactional or not, default is *false*. If the policy is transactional and the update policy fails, the source table isn't updated. |
110112
|PropagateIngestionProperties |`bool`|States if properties specified during ingestion to the source table, such as [extent tags](extent-tags.md) and creation time, apply to the target table. |
@@ -144,6 +146,14 @@ Update policies take effect when data is ingested or moved to a source table, or
144146
> Data may be lost in all tables with an update policy relationship if the `replace` command is invoked.
145147
> Consider using `.set-or-append` instead.
146148
149+
## Update policy with source table wildcard pattern
150+
151+
Update policy supports ingesting from multiple source tables that share the same pattern, while using the
152+
same query as the update policy query. This is useful if you have several source tables, usually sharing the same schema (or a subset of columns that share a common schema), and you would like to trigger ingestion to a
153+
single target table, when ingesting to either of those tables. In this case, instead of defining multiple
154+
update policies, each for a single source table, you can define a single update policy with wildcard as `Source`.The `Query` of the update policy must comply with all source tables matching the pattern.
155+
To reference the source table in the update policy query, you can use a special symbol named `$source_table`. See example in [Example of wild card update policy](#example-of-wild-card-update-policy).
156+
147157
## Remove data from source table
148158

149159
After ingesting data to the target table, you can optionally remove it from the source table. Set a soft-delete period of `0sec` (or `00:00:00`) in the source table's [retention policy](retention-policy.md), and the update policy as transactional. The following conditions apply:
@@ -251,6 +261,86 @@ In this example, use an update policy with a simple function to perform ETL. Fir
251261
.alter-merge table MySourceTable policy retention softdelete = 0s
252262
```
253263
264+
## Example of wild card update policy
265+
266+
The following example creates an update policy with a single entry on table `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source.
267+
Any ingestion to a table which matches the pattern (in local database) will trigger the update policy, and ingest data to `TargetTable`, based on the update policy query.
268+
269+
1. Create two source tables:
270+
271+
```kusto
272+
.create table SourceTable1(Id:long, Value:string)
273+
```
274+
275+
```kusto
276+
.create table SourceTable2(Id:long, Value:string)
277+
```
278+
279+
1. Create the target table:
280+
281+
```kusto
282+
.create table TargetTable(Id:long, Value:string, Source:string)
283+
```
284+
285+
1. Create a function which will serve as the `Query` of the update policy. The function uses the `$source_table` symbol to reference the `Source` of the update policy. Use `skipValidation=true` to skip validation during the create function, since `$source_table` is only known during update policy execution. The function is validated during the next step, when altering the update policy.
286+
287+
```kusto
288+
.create function with(skipValidation=true) IngestToTarget()
289+
{
290+
$source_table
291+
| parse Value with "I'm from table " Source
292+
| project Id, Value, Source
293+
}
294+
```
295+
296+
1. Create the update policy on `TargetTable`. The policy references all tables matching pattern `SourceTable*` as its source.
297+
298+
````kusto
299+
.alter table TargetTable policy update
300+
```[{
301+
"IsEnabled": true,
302+
"Source": "SourceTable*",
303+
"SourceIsWildCard" : true,
304+
"Query": "IngestToTarget()",
305+
"IsTransactional": true,
306+
"PropagateIngestionProperties": true
307+
}]```
308+
309+
````
310+
311+
1. Ingest to source tables. Both ingestions trigger the update policy:
312+
313+
```kusto
314+
.set-or-append SourceTable1 <|
315+
datatable (Id:long, Value:string)
316+
[
317+
1, "I'm from table SourceTable1",
318+
2, "I'm from table SourceTable1"
319+
]
320+
```
321+
322+
```kusto
323+
.set-or-append SourceTable2 <|
324+
datatable (Id:long, Value:string)
325+
[
326+
3, "I'm from table SourceTable2",
327+
4, "I'm from table SourceTable2"
328+
]
329+
```
330+
331+
1. Query `TargetTable`:
332+
333+
```kusto
334+
TargetTable
335+
```
336+
337+
|Id|Value|Source|
338+
|---|---|---|
339+
|1|I'm from table SourceTable1|SourceTable1|
340+
|2|I'm from table SourceTable1|SourceTable1|
341+
|3|I'm from table SourceTable2|SourceTable2|
342+
|4|I'm from table SourceTable2|SourceTable2|
343+
254344
## Related content
255345
256346
* [Common scenarios for using table update policies](update-policy-common-scenarios.md)

0 commit comments

Comments
 (0)