[Streams] Update data retention to include frozen tiers for DSL streams - #7577
[Streams] Update data retention to include frozen tiers for DSL streams#7577mdbirnstiehl wants to merge 6 commits into
Conversation
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🔍 Preview links for changed docs |
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Docs review summary
Focus areas
- Style and clarity: Vale reported no findings; manual read found no additional high-confidence issues. Wording changes (e.g., "choose" → "select", "predefined") align with the Elastic word-choice guide.
- Jargon: No unexplained Elastic-internal jargon introduced. "DSL" is expanded on first use, and
{{ilm-init}}/{{search-snaps}}use existing substitutions. - Frontmatter and applies_to: Frontmatter unchanged and valid. New inline
{applies_to}and block{applies_to}tags follow the repo's established syntax and version format. - Content type fit: How-to structure preserved; new steps are consistent with existing procedure style.
- Parent issue satisfaction: Partially satisfied. #7201 (frozen phase in the Data lifecycle timeline, tier-split ingestion/storage) is covered by this PR. #7283 (Enterprise license requirement and gating modal for the frozen phase) is only partially covered — the snapshot-repository requirement is documented, but the Enterprise license requirement and the "hidden when no repository and no privilege" behavior are not mentioned. See inline comment.
Notes
- The PR body itself flags that
applies_totagging for this content still needs more thought — the reviewer should confirm thestack: ga 9.5+values with the feature owner before merge.
Generated by Docs review agent for #7577 · sonnet50 45.6 AIC · ⌖ 3.85 AIC · ⊞ 15.6K
| : The index is no longer updated and is queried rarely. Optimized for long-term retention at the lowest possible cost. Set the minimum age for data to move into this phase and configure a snapshot repository. The frozen phase requires a snapshot repository. | ||
| : The index is no longer updated and is queried rarely. Optimized for long-term retention at the lowest possible cost. Set the minimum age for data to move into this phase and configure a snapshot repository. | ||
|
|
||
| {applies_to}`stack: ga 9.5+` For streams using a DSL, you need to set a default snapshot repository before adding a frozen phase. If no default repository is set, you'll be prompted to set one. After setting it, select {icon}`refresh` to resume. |
There was a problem hiding this comment.
The linked issue #7283 also calls for documenting the Enterprise license requirement for the frozen phase (badge shown when missing, gating modal prompting an upgrade). This paragraph only covers the snapshot-repository requirement. Consider adding a sentence noting that adding a Frozen phase requires an Enterprise license, and that the option is hidden if there is no default repository and the user lacks privileges to create one.
florent-leborgne
left a comment
florent-leborgne
left a comment
There was a problem hiding this comment.
I didn't comment on all instances on this page but it looks to me that this page is no longer correct or lacks indications for anyone not on the latest version. Likely misses a few applies_to or name alternatives (that we can do without applies_to to keep simple like in my 1st suggestion)
damian-polewski
left a comment
damian-polewski
left a comment
There was a problem hiding this comment.
Thank you @mdbirnstiehl for working on this, it looks great!
I added a couple of comments with the things I noticed. Let me know what do you think!
| # Configure data retention with Streams [streams-configure-retention] | ||
|
|
||
| Managing data retention across multiple indexes typically requires configuring {{ilm}} ({{ilm-init}}), data stream lifecycle, index templates, and index settings, each in a different place. Streams replaces this with a single UI so you can control storage and meet regulatory or compliance requirements. | ||
| Managing data retention across multiple indexes typically requires configuring {{ilm}} ({{ilm-init}}), data stream lifecycle (DSL), index templates, and index settings, each in a different place. Streams replaces this with a single UI so you can control storage and meet regulatory or compliance requirements. |
There was a problem hiding this comment.
Do we want to keep it consistent and use lifecycle here instead of retention?
| Managing data retention across multiple indexes typically requires configuring {{ilm}} ({{ilm-init}}), data stream lifecycle (DSL), index templates, and index settings, each in a different place. Streams replaces this with a single UI so you can control storage and meet regulatory or compliance requirements. | |
| Managing data lifecycle across multiple indexes typically requires configuring {{ilm}} ({{ilm-init}}), data stream lifecycle (DSL), index templates, and index settings, each in a different place. Streams replaces this with a single UI so you can control storage and meet regulatory or compliance requirements. |
Also one question to @EdLewisEL we are still using DSL here but I remember we talked about referring to data stream lifecycle as DLM. Should we update the documentation to include that change?
| Before setting a retention policy, review the following panels to understand your data's footprint: | ||
|
|
||
| - **Storage size**: Total data volume and document count for the stream. | ||
| - **Storage size**: Total data volume and document count for the stream, including data across all tiers. |
There was a problem hiding this comment.
@EdLewisEL I remember some discussion during development that ES wanted to stop using tiers and move to phases, because tiers can be ambiguous. Was that true?
| - **Storage size**: Total data volume and document count for the stream, including data across all tiers. | |
| - **Storage size**: Total data volume and document count for the stream, including data across all phases. |
4 participants
Summary
This PR closes #7283 and #7201 and adds the ability to set frozen and delete phases in streams using DSL data retention.
Some thought still needs to go into how to organize this data using
applies_totaggingGenerative AI disclosure
Tool(s) and model(s) used: Claude Code and Sonnet 4.6