Skip to content

Commit bb80304

Browse files
author
Kostiantyn Krykun
committed
docs: Add Tekton Results to docs
1 parent d7154e8 commit bb80304

2 files changed

Lines changed: 54 additions & 118 deletions

File tree

‎cspell.config.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,7 @@ words:
189189
- petclinic
190190
- pgcrypto
191191
- pgdb
192+
- pgbackrest
192193
- PGPASSWORD
193194
- pipelinerun
194195
- pipelineruns

‎docs/operator-guide/ci/tekton-long-term-storage.md‎

Lines changed: 53 additions & 118 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
title: "Tekton Long-Term Log Storage"
3-
description: "Learn how to configure and access long-term logs for Tekton pipelines in KubeRocketCI using OpenSearch, enhancing log management and observability."
4-
sidebar_label: "Tekton Long-Term Log Storage"
3+
description: "Tekton Results helps you logically group CI/CD workload history and separate long-term result storage from the Pipeline controller: add custom Result metadata (e.g. post-run actions), group related TaskRuns and PipelineRuns, keep result history independent of the Pipeline CRD controller to free etcd resources, and store logs so completed Runs can be cleaned."
4+
sidebar_label: "Tekton Results Integration"
55
---
66
<!-- markdownlint-disable MD025 -->
77

@@ -14,151 +14,86 @@ import TabItem from '@theme/TabItem';
1414
<link rel="canonical" href="https://docs.kuberocketci.io/docs/operator-guide/ci/tekton-long-term-storage" />
1515
</head>
1616

17-
KubeRocketCI Portal provides the capability to view long-term logs for pipelines. The long-term logs are stored in the OpenSearch cluster and can be viewed in the KubeRocketCI Portal after pipeline cleanup or deletion. This guide describes how to configure and view long-term logs in the KubeRocketCI Portal.
17+
Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller. This allows you to:
18+
19+
- Provide custom Results metadata about your CI/CD workflows not available in the Tekton TaskRun/PipelineRun CRDs (for example: post-run actions).
20+
- Group related workloads together (e.g. bundle related TaskRuns and PipelineRuns into a single unit).
21+
- Make long-term result history independent of the Pipeline CRD controller, letting you free up etcd resources for Run execution.
22+
- Store logs produced by the TaskRuns/PipelineRuns so that completed Runs can be cleaned to save resources.
1823

1924
## Long-Term Log Access Workflow
2025

2126
The following diagram illustrates the workflow for accessing long-term logs for pipelines in the KubeRocketCI Portal:
2227

28+
2329
```mermaid
2430
sequenceDiagram
2531
actor U as User
2632
participant KR as KubeRocketCI Portal
2733
participant PC as Pipeline Controller
28-
participant RW as Fluent Bit Agent
29-
participant KD as KrakenD API Gateway
30-
participant RA as OpenSearch
34+
participant RW as Result Watcher
35+
participant RA as Result API
3136
32-
U->>KR: Trigger or create a pipeline
37+
U->>KR: Trigger or create a PipelineRun/TaskRun
3338
KR->>PC: Start pipeline execution
34-
RW-->>PC: Collect pipeline logs
35-
RW->>RA: Send logs to OpenSearch
36-
Note over KR,RA: Pipeline is deleted or cleaned up
39+
RW-->>PC: Watch PipelineRun/TaskRun
40+
Note over PC,RW: Wait for PipelineRun/TaskRun Completion
41+
RW->>RA: Update results database
3742
U->>KR: View pipeline logs
38-
KR-->>KD: Request long-term logs
39-
KD-->>RA: Fetch long-term logs
43+
KR-->>RA: Get long-term logs
4044
KR->>U: Return long-term logs
45+
4146
```
4247

43-
## Configuration
44-
45-
To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:
46-
47-
1. Install and configure OpenSearch cluster:
48-
49-
:::note
50-
To install OpenSearch cluster, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/opensearch) repository, which contains already prepared configurations for OpenSearch installation.
51-
:::
52-
53-
To install OpenSearch cluster using the add-ons repository, follow the steps below:
54-
55-
1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
56-
57-
2. Navigate to the `clusters/core/addons/opensearch` directory and configure the `values.yaml` file with the necessary values for OpenSearch cluster installation.
58-
59-
:::note
60-
Ensure that [Fluent Bit](https://github.com/epam/edp-cluster-add-ons/blob/44ca88c079d464c826fcae38f3f03fe983d1f984/clusters/core/addons/opensearch/values.yaml#L391) is configured to send container logs to the `logstash-edp` index in the OpenSearch cluster, as the KubeRocketCI Portal uses this index to retrieve long-term logs.
61-
:::
62-
63-
3. Install the OpenSearch cluster using Helm or [Argo CD](../add-ons-overview.md) methods.
64-
65-
2. Install and configure KrakenD API Gateway:
66-
67-
:::note
68-
To install KrakenD API Gateway with OpenSearch connection, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend) repository, which contains already prepared configurations for KrakenD installation.
69-
For more details, refer to the [KrakenD Integration](../extensions/krakend.md) guide.
70-
:::
71-
72-
To install KrakenD API Gateway with OpenSearch connection, follow the steps below:
73-
74-
1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
75-
76-
2. Navigate to the `clusters/core/addons/krakend` directory and configure the `values.yaml` file with the necessary values for KrakenD installation.
77-
78-
:::note
79-
Ensure that the KrakenD endpoint object for OpenSearch has the `/search/logs` endpoint name, as the KubeRocketCI Portal uses this endpoint to fetch long-term logs.
80-
:::
81-
82-
3. Ensure that the KrakenD [configuration secret](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/krakend/templates/external-secrets/krakend.yaml) contains the OpenSearch connection variables.
48+
## Install Tekton Results
8349

84-
<Tabs
85-
defaultValue="externalsecret"
86-
values={[
87-
{label: 'Manifests', value: 'manifests'},
88-
{label: 'External Secrets Operator', value: 'externalsecret'},
89-
]}>
50+
Tekton Results is deployed as part of the Tekton Pipelines installation. For installing Tekton Pipelines, we recommend using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/blob/4456631feb6510ff875b5b839534968c3846da7a/clusters/core/apps/values.yaml#L286C1-L289C30). Here's an example of how to configure values.yaml before installation:
9051

91-
<TabItem value="manifests">
52+
```yaml
53+
tekton:
54+
createNamespace: true
55+
enable: true
56+
namespace: tekton-pipelines
57+
```
9258

93-
```yaml
94-
apiVersion: v1
95-
kind: Secret
96-
metadata:
97-
name: krakend
98-
namespace: krakend
99-
type: Opaque
100-
stringData:
101-
OPENSEARCH_URL: https://opensearch-cluster-master.logging:9200
102-
OPENSEARCH_CREDS: <base64-encoded-credentials>
103-
```
104-
105-
</TabItem>
106-
107-
<TabItem value="externalsecret">
108-
109-
```json
110-
{
111-
"OPENSEARCH_URL": "https://opensearch-cluster-master.logging:9200",
112-
"OPENSEARCH_CREDS": "<base64-encoded-credentials>"
113-
}
114-
```
115-
</TabItem>
116-
117-
</Tabs>
118-
119-
4. Install the KrakenD API Gateway using Helm or [Argo CD](../add-ons-overview.md) methods.
120-
121-
## Viewing Long-Term Logs
122-
123-
After configuring long-term log storage for Pipelines in the KubeRocketCI Portal, follow the steps below to view long-term logs:
124-
125-
1. Navigate to the KubeRocketCI Portal and sign in with appropriate credentials.
126-
127-
![KubeRocketCI Portal](../../assets/operator-guide/ci/kuberocketci-portal.png "KubeRocketCI Portal")
128-
129-
2. In the **Components** section, navigate to the appropriate component (if component is not created yet, refer to the [Add Application](../../user-guide/add-application.md) guide to create a new one).
130-
131-
![KubeRocketCI Portal Components](../../assets/operator-guide/ci/portal-components.png "KubeRocketCI Portal Components")
132-
133-
3. In the component window, in the right upper corner, click the **GIT** button to navigate to the component repository (e.g., GitHub repository).
134-
135-
![Component Git Repository](../../assets/operator-guide/ci/component-git-repository.png "Component Git Repository")
136-
137-
4. In the component repository, create a Pull Request to trigger the review pipeline execution.
138-
139-
![Create Pull Request](../../assets/operator-guide/ci/create-pull-request.png "Create Pull Request")
140-
141-
5. In the Pull Request checks section, click the **Show all checks** button and navigate to the review pipeline in the KubeRocketCI Portal by clicking the **Details** button.
59+
## Configuration
14260

143-
![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
61+
To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:
14462

145-
6. In the pipeline window, check the pipeline status. Ensure that the pipeline is successfully executed and the logs are available.
63+
1. Storage sizes in the [results-pg.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results-pg.yaml) file define the allocated volumes for the Tekton Results PostgreSQL cluster:
14664

147-
![Pipeline Logs](../../assets/operator-guide/ci/pipeline-logs.png "Pipeline Logs")
65+
- **Database instance storage**
66+
- Path: `spec.instances.dataVolumeClaimSpec.resources.requests.storage`
67+
- Default: `2Gi`
68+
- Description: Specifies the size of the main PostgreSQL data volume.
14869

149-
7. Navigate to the **Pipelines** section and find the executed review pipeline. Delete the pipeline by clicking the **Delete** button.
70+
- **Backup repository storage**
71+
- Path: `spec.backups.pgbackrest.repos.volume.volumeClaimSpec.resources.requests.storage`
72+
- Default: `2Gi`
73+
- Description: Specifies the size of the pgBackRest backup repository volume.
15074

151-
![Delete Pipeline](../../assets/operator-guide/ci/delete-pipeline.png "Delete Pipeline")
75+
2. Storage and retention settings in the [results.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results.yaml) file define how Tekton Results logs and data are stored and managed:
15276

153-
8. Return to the Pull Request and click the **Show all checks** button. Navigate to the review pipeline again by clicking the **Details** button.
77+
- **Log storage size**
78+
- Path: `PersistentVolumeClaim.spec.resources.requests.storage`
79+
- Default: `5Gi`
80+
- Description: Specifies the allocated disk size for storing pipeline logs.
15481

155-
![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
82+
- **Retention period**
83+
- Path: `ConfigMap.data.defaultRetention` in `tekton-results-config-results-retention-policy`
84+
- Default: `"30"` days
85+
- Description: Defines how long log files are retained before being automatically removed.
15686

157-
9. After the pipeline deletion, the long-term logs are displayed in the pipeline window.
87+
- **Cleanup schedule**
88+
- Path: `ConfigMap.data.runAt` in `tekton-results-config-results-retention-policy`
89+
- Default: `"5 5 * * 0"`
90+
- Description: Specifies the cron expression that determines when automated log cleanup runs (every Sunday at 5:05 AM).
15891

159-
![Long Term Logs](../../assets/operator-guide/ci/reserved-logs.png "Long Term Logs")
92+
:::warning
93+
The retention policy agent removes only database records from PostgreSQL. It does not delete associated log files stored on persistent volumes, S3, or GCS backends. As a result, log data may remain on disk and consume storage even after the corresponding records are expired.
16094

161-
This example demonstrates how to view long-term logs for review pipeline, but the same approach can be applied to other pipeline types.
95+
Users are currently responsible for manually cleaning up unused log files from the underlying storage backends.
96+
:::
16297

16398
## Related Articles
16499

0 commit comments

Comments
 (0)