Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
763 changes: 605 additions & 158 deletions Cargo.lock

Large diffs are not rendered by default.

9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Core Dump Handler

This helm chart is designed to deploy functionality that automatically saves core dumps from most public cloud kubernetes service providers and private kubernetes instances to an S3 compatible storage service.
This helm chart is designed to deploy functionality that automatically saves core dumps from most public cloud kubernetes service providers and private kubernetes instances to an object storage service.

[![build status](https://github.com/ibm/core-dump-handler/workflows/CI/badge.svg)](https://github.com/ibm/core-dump-handler/actions)
[![Docker Repository on Quay](https://quay.io/repository/icdh/core-dump-handler/status "Docker Repository on Quay")](https://quay.io/repository/icdh/core-dump-handler)
Expand Down Expand Up @@ -99,7 +99,7 @@ This chart aims to tackle the problems surrounding core dumps by leveraging comm

The chart deploys two processes:

1. The **agent** manages the updating of `/proc/sys/kernel/*` configuration, deploys the composer service and uploads the core dumps zipfile created by the composer to an object storage instance.
1. The **agent** manages the updating of `/proc/sys/kernel/*` configuration, deploys the composer service and uploads the core dumps zipfile created by the composer to an object storage instance. Native S3-compatible and Azure Blob Storage uploads are supported.

2. The **composer** handles the processing of a core dump and creating runtime, container coredump and image JSON documents from CRICTL and inserting them into a single zip file. The zip file is stored on the local file system of the node for the agent to upload.

Expand Down Expand Up @@ -205,6 +205,11 @@ or run the helm install command with the settings
S3_SECRET=XXXX
S3_BUCKET_NAME=XXXX
S3_REGION=XXXX

# Or for Azure Blob Storage
STORAGE_PROVIDER=azure
AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=XXXX;AccountKey=XXXX;EndpointSuffix=core.windows.net
AZURE_STORAGE_CONTAINER_NAME=core-dumps
```

1. Change directory to the integration folder and run the test
Expand Down
59 changes: 56 additions & 3 deletions charts/core-dump-handler/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Core Dump Handler

This helm chart is designed to deploy functionality that automatically saves core dumps from any public cloud kubernetes service provider or [RedHat OpenShift Kubernetes Service](https://cloud.ibm.com/kubernetes/catalog/create?platformType=openshift) to an S3 compatible storage service.
This helm chart is designed to deploy functionality that automatically saves core dumps from any public cloud kubernetes service provider or [RedHat OpenShift Kubernetes Service](https://cloud.ibm.com/kubernetes/catalog/create?platformType=openshift) to an object storage service.

## Prerequisites

The [Helm](https://helm.sh/) cli to run the chart

An [S3 Protocol Compatible](https://en.wikipedia.org/wiki/Amazon_S3) object storage solution.
An object storage solution reachable from the cluster. Native S3-compatible uploads and Azure Blob Storage uploads are supported.

A [CRIO](https://cri-o.io/) compatible container runtime on the kubernetes hosts. If you service provider uses something else we will willingly recieve patches to support them.

Expand All @@ -22,6 +22,30 @@ helm install core-dump-handler . --create-namespace --namespace observe \

Where the `--set` options are configuration for your S3 protocol compatible provider

Azure Blob Storage is auto-detected when `daemonset.azureClientId` is present. The agent prioritizes managed identity authentication; if the client ID is not set, it falls back to connection string / account key.

**With Azure Workload Identity (recommended):**

```
helm install core-dump-handler . --create-namespace --namespace observe \
--set daemonset.azureClientId=00000000-0000-0000-0000-000000000000 \
--set daemonset.azureTenantId=00000000-0000-0000-0000-000000000000 \
--set daemonset.azureStorageContainerName=core-dumps \
--set daemonset.azureStorageBlobEndpoint=https://myaccount.blob.core.windows.net \
--set serviceAccount.annotations.azure\.workload\.identity/client-id=00000000-0000-0000-0000-000000000000 \
--set daemonset.podLabels.azure\.workload\.identity/use=true
```

Note: `storageProvider` is automatically set to Azure when a client ID is provided.

**Fallback with connection string (if managed identity is not available):**

```
helm install core-dump-handler . --create-namespace --namespace observe \
--set daemonset.azureStorageConnectionString='DefaultEndpointsProtocol=https;AccountName=XXX;AccountKey=XXX;EndpointSuffix=core.windows.net' \
--set daemonset.azureStorageContainerName=core-dumps
```

For the following providers an additional option of values should be provided using the `--values` flag

e.g.
Expand All @@ -33,6 +57,9 @@ helm install core-dump-handler . --create-namespace --namespace observe \
```

<table><thead><td>Provider</td><td>Product</td><td>Values</td></thead>
<tr>
<td>Microsoft</td><td>AKS with Azure Blob Storage</td><td><a href="values.azure.yaml">values.azure.yaml</a></td>
</tr>
<tr>
<td>AWS</td><td>EKS</td><td><a href="values.aws.yaml">values.aws.yaml</a></td>
</tr>
Expand Down Expand Up @@ -234,6 +261,13 @@ The agent pod has the following environment variables and these are all set by t
e.g. --set S3_REGION=host.mycloud.com

See https://github.com/IBM/core-dump-handler/issues/124 for further discussion.
* STORAGE_PROVIDER - Optional explicit backend selector. Supported values are `s3` and `azure`. When omitted, the agent auto-detects Azure if Azure storage settings are present, otherwise it uses S3.
* AZURE_STORAGE_CONNECTION_STRING - Azure Blob Storage connection string. When set, the agent derives account and endpoint settings from it.
* AZURE_STORAGE_ACCOUNT_NAME - Azure storage account name when not using a connection string.
* AZURE_STORAGE_ACCOUNT_KEY - Azure storage account key when not using a connection string.
* AZURE_STORAGE_CONTAINER_NAME - Blob container that receives the uploaded archives.
* AZURE_STORAGE_BLOB_ENDPOINT - Blob service endpoint, for example `https://myaccount.blob.core.windows.net`.
* AZURE_STORAGE_BLOB_PREFIX - Optional virtual folder prefix prepended to each uploaded blob name.
* VENDOR - Some older hosts may require targeted builds for the composer.

default(Default) - A RHEL8 build
Expand All @@ -258,6 +292,18 @@ The following secrets are configurable and map to the corresponding environment

key: s3Region

key: azureStorageConnectionString

key: azureStorageAccountName

key: azureStorageAccountKey

key: azureStorageContainerName

key: azureStorageBlobEndpoint

key: azureStorageBlobPrefix

### Values

General
Expand Down Expand Up @@ -322,14 +368,21 @@ Daemonset
* useINotify: Maps to the USE_INOTIFY environment variable (Default false)
* DeployCrioConfig: Maps to the DEPLOY_CRIO_CONFIG enviroment variable (Default false)
* includeCrioExe: Maps to the DEPLOY_CRIO_EXE enviroment variable (Default false)
* manageStoreSecret: Defines if the chart will be responsible for creating the S3 environment variables.
* manageStoreSecret: Defines if the chart will be responsible for creating the storage environment variables.

Set to false if you are using an external secrets managment system (Default true)

* storageProvider : Maps to the STORAGE_PROVIDER enviroment variable. Use `azure` to force Azure Blob Storage, `s3` to force S3, or leave empty for autodetection.
* s3AccessKey : Maps to the S3_ACCESS_KEY enviroment variable
* s3Secret : Maps to the S3_SECRET enviroment variable
* s3BucketName : Maps to the S3_BUCKET_NAME enviroment variable
* 3Region : Maps to the S3_REGION enviroment variable
* azureStorageConnectionString : Maps to the AZURE_STORAGE_CONNECTION_STRING environment variable
* azureStorageAccountName : Maps to the AZURE_STORAGE_ACCOUNT_NAME environment variable
* azureStorageAccountKey : Maps to the AZURE_STORAGE_ACCOUNT_KEY environment variable
* azureStorageContainerName : Maps to the AZURE_STORAGE_CONTAINER_NAME environment variable
* azureStorageBlobEndpoint : Maps to the AZURE_STORAGE_BLOB_ENDPOINT environment variable
* azureStorageBlobPrefix : Maps to the AZURE_STORAGE_BLOB_PREFIX environment variable
* extraEnvVars: Option for passing additional configuration to the agent such as endpoint properties.
* envFrom: Array of [EnvFromSource](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#envfromsource-v1-core) to inject into main container.
* sidecarContainers: Array of [Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#container-v1-core) to define as part of the pod.
Expand Down
51 changes: 51 additions & 0 deletions charts/core-dump-handler/templates/daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ spec:
metadata:
labels:
name: {{ .Values.daemonset.label }}
{{- with .Values.daemonset.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
kubectl.kubernetes.io/default-container: "coredump-container"
{{- with .Values.daemonset.podAnnotations }}
Expand Down Expand Up @@ -75,6 +78,16 @@ spec:
value: {{ .Values.daemonset.suidDumpable | quote }}
- name: DEPLOY_CRIO_EXE
value: {{ .Values.daemonset.includeCrioExe | quote }}
- name: STORAGE_PROVIDER
value: {{ .Values.daemonset.storageProvider | quote }}
{{- if .Values.daemonset.azureClientId }}
- name: AZURE_CLIENT_ID
value: {{ .Values.daemonset.azureClientId | quote }}
{{- end }}
{{- if .Values.daemonset.azureTenantId }}
- name: AZURE_TENANT_ID
value: {{ .Values.daemonset.azureTenantId | quote }}
{{- end }}
{{- if .Values.daemonset.manageStoreSecret }}
- name: S3_ACCESS_KEY
valueFrom:
Expand All @@ -93,11 +106,49 @@ spec:
secretKeyRef:
name: s3config
key: s3BucketName
optional: true
- name: S3_REGION
valueFrom:
secretKeyRef:
name: s3config
key: s3Region
optional: true
- name: AZURE_STORAGE_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: s3config
key: azureStorageConnectionString
optional: true
- name: AZURE_STORAGE_ACCOUNT_NAME
valueFrom:
secretKeyRef:
name: s3config
key: azureStorageAccountName
optional: true
- name: AZURE_STORAGE_ACCOUNT_KEY
valueFrom:
secretKeyRef:
name: s3config
key: azureStorageAccountKey
optional: true
- name: AZURE_STORAGE_CONTAINER_NAME
valueFrom:
secretKeyRef:
name: s3config
key: azureStorageContainerName
optional: true
- name: AZURE_STORAGE_BLOB_ENDPOINT
valueFrom:
secretKeyRef:
name: s3config
key: azureStorageBlobEndpoint
optional: true
- name: AZURE_STORAGE_BLOB_PREFIX
valueFrom:
secretKeyRef:
name: s3config
key: azureStorageBlobPrefix
optional: true
{{- end }}
- name: VENDOR
value: {{ .Values.daemonset.vendor }}
Expand Down
22 changes: 22 additions & 0 deletions charts/core-dump-handler/templates/secrets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,28 @@ stringData:
{{- if .Values.daemonset.s3AccessKey }}
s3AccessKey: {{ .Values.daemonset.s3AccessKey }}
{{- end }}
{{- if .Values.daemonset.s3BucketName }}
s3BucketName: {{ .Values.daemonset.s3BucketName }}
{{- end }}
{{- if .Values.daemonset.s3Region }}
s3Region: {{ .Values.daemonset.s3Region }}
{{- end }}
{{- if .Values.daemonset.azureStorageConnectionString }}
azureStorageConnectionString: {{ .Values.daemonset.azureStorageConnectionString }}
{{- end }}
{{- if .Values.daemonset.azureStorageAccountName }}
azureStorageAccountName: {{ .Values.daemonset.azureStorageAccountName }}
{{- end }}
{{- if .Values.daemonset.azureStorageAccountKey }}
azureStorageAccountKey: {{ .Values.daemonset.azureStorageAccountKey }}
{{- end }}
{{- if .Values.daemonset.azureStorageContainerName }}
azureStorageContainerName: {{ .Values.daemonset.azureStorageContainerName }}
{{- end }}
{{- if .Values.daemonset.azureStorageBlobEndpoint }}
azureStorageBlobEndpoint: {{ .Values.daemonset.azureStorageBlobEndpoint }}
{{- end }}
{{- if .Values.daemonset.azureStorageBlobPrefix }}
azureStorageBlobPrefix: {{ .Values.daemonset.azureStorageBlobPrefix }}
{{- end }}
{{- end }}
19 changes: 19 additions & 0 deletions charts/core-dump-handler/values.azure.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Azure Blob Storage with Managed Identity (recommended)
#
# When AZURE_CLIENT_ID is set, the agent automatically selects Azure Blob Storage
# and authenticates using the managed identity. No need to set storageProvider.

daemonset:
manageStoreSecret: true
azureClientId: "00000000-0000-0000-0000-000000000000"
azureTenantId: "00000000-0000-0000-0000-000000000000"
azureStorageContainerName: core-dumps
azureStorageBlobEndpoint: "https://XXXX.blob.core.windows.net"

# Required for Azure Workload Identity on AKS.
podLabels:
azure.workload.identity/use: "true"

serviceAccount:
annotations:
azure.workload.identity/client-id: "00000000-0000-0000-0000-000000000000"
84 changes: 80 additions & 4 deletions charts/core-dump-handler/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -174,18 +174,61 @@
"properties": {
"manageStoreSecret": {
"const": true
},
"storageProvider": {
"const": "azure"
}
},
"required": [
"manageStoreSecret"
"manageStoreSecret",
"storageProvider"
]
},
"then": {
"required": [
"azureStorageContainerName"
],
"anyOf": [
{
"required": [
"azureStorageConnectionString"
]
},
{
"required": [
"azureStorageAccountName",
"azureStorageAccountKey",
"azureStorageBlobEndpoint"
]
}
]
}
},
{
"if": {
"properties": {
"manageStoreSecret": {
"const": true
}
},
"required": [
"manageStoreSecret"
],
"not": {
"properties": {
"storageProvider": {
"const": "azure"
}
},
"required": [
"storageProvider"
]
}
},
"then": {
"required": [
"s3AccessKey",
"s3BucketName",
"s3Region",
"s3Secret"
"s3Region"
]
}
},
Expand Down Expand Up @@ -215,6 +258,12 @@
"type": "string"
}
},
"podLabels": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"hostDirectory": {
"type": "string"
},
Expand All @@ -230,6 +279,15 @@
"vendor": {
"type": "string"
},
"storageProvider": {
"type": "string"
},
"azureClientId": {
"type": "string"
},
"azureTenantId": {
"type": "string"
},
"interval": {
"type": "integer",
"minimum": 1000
Expand Down Expand Up @@ -270,6 +328,24 @@
"s3Region": {
"type": "string"
},
"azureStorageConnectionString": {
"type": "string"
},
"azureStorageAccountName": {
"type": "string"
},
"azureStorageAccountKey": {
"type": "string"
},
"azureStorageContainerName": {
"type": "string"
},
"azureStorageBlobEndpoint": {
"type": "string"
},
"azureStorageBlobPrefix": {
"type": "string"
},
"extraEnvVars": {
"type": "string"
},
Expand Down
Loading