VED-1233: Replace manual release steps#1437
Open
Thomas-Boyle wants to merge 9 commits intomasterfrom
Open
Conversation
- Added steps to set the Terraform workspace and manage shared Lambda triggers during blue/green deployments in the deploy-backend.yml workflow. - Introduced a new script, manage_blue_green_event_source_mappings.sh, to handle the preparation and cleanup of event source mappings for Lambda functions. - Updated README.md to document the new blue/green Lambda trigger handoff process, removing manual steps from the deployment flow.
Contributor
|
This branch is working on a ticket in the NHS England VED JIRA Project. Here's a handy link to the ticket: VED-1233 |
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
added
feature
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev-sandbox
GitHub Actions
Inactive
Thomas-Boyle
temporarily deployed
to
internal-dev
GitHub Actions
Inactive
…event_source_mappings.sh - Introduced a new delete_mapping function to handle the deletion of AWS Lambda event source mappings, including a timeout mechanism for deletion confirmation. - Updated adopt_mapping function to utilize the new delete_mapping function, improving the logic for handling target and counterpart mapping UUIDs. - Enhanced code clarity and maintainability by restructuring the mapping lookup and deletion process.
avshetty1980
reviewed
May 5, 2026
| @@ -0,0 +1,170 @@ | |||
| #!/usr/bin/env bash | |||
Contributor
There was a problem hiding this comment.
Can this script delete a live lambda event source mapping before terraform apply - outside any saved plan?
This should only be used for the controlled migration. If this script fails between the adopt and apply then state and/or AWS can get out of sync which will not be recorded in the artifact.
Can we move this to a dedicated migration workflow (or behind a one-time flag per env)?
avshetty1980
reviewed
May 5, 2026
avshetty1980
reviewed
May 5, 2026
avshetty1980
reviewed
May 5, 2026
avshetty1980
reviewed
May 5, 2026
avshetty1980
reviewed
May 5, 2026
avshetty1980
reviewed
May 5, 2026
avshetty1980
reviewed
May 5, 2026
Contributor
avshetty1980
left a comment
avshetty1980
left a comment
There was a problem hiding this comment.
Looking really good - just a few questions and comments.
- Added a new workflow for migrating event source mappings, allowing controlled one-time migrations for specific environments. - Updated the deploy-backend.yml workflow to include concurrency settings and additional steps for Terraform initialization, formatting, validation, and applying event source mappings. - Refactored the Makefile to introduce new commands for formatting checks, validation, and applying Terraform plans. - Enhanced the adopt_event_source_mappings.sh script to support verification of event source mappings and improved logging for existing mappings. - Updated README.md to document the new migration process and rollback procedures for event source mappings.
avshetty1980
reviewed
May 7, 2026
| @@ -110,6 +110,10 @@ env: # Sonarcloud - do not allow direct usage of untrusted data | |||
|
|
|||
| run-name: Deploy Backend - ${{ inputs.environment }} ${{ inputs.sub_environment }} | |||
Contributor
There was a problem hiding this comment.
Would it be worth adding a validate step for the event source mapping module (maybe with -backend=false) and a shellcheck step for the adopt script in the quality-checks.yml workflow?
avshetty1980
reviewed
May 7, 2026
| sub_environment ?= $(SUB_ENVIRONMENT) | ||
| sub_environment_dir := $(if $(findstring pr-,$(sub_environment)),pr,$(sub_environment)) | ||
| tf_var_file := ../instance/environments/$(environment)/$(sub_environment_dir)/variables.tfvars | ||
| has_sub_environment_scope = $(shell awk -F= '/^has_sub_environment_scope/ { gsub(/[[:space:]]/, "", $$2); print $$2 }' "$(tf_var_file)") |
Contributor
There was a problem hiding this comment.
if awk returns empty then workspace_name becomes $(sub_environment). This would mean separate workspace for blue and green rather than a shared one - which would defeat the duplicating state idea.Should we add a guard here eg:
ifeq ($(has_sub_environment_scope),)
$(error has_sub_environment_scope not found in $(tf_var_file))
endif
avshetty1980
reviewed
May 7, 2026
|
|
||
| tf_state = \ | ||
| -backend-config="bucket=$(bucket_name)" \ | ||
| -backend-config="key=event-source-mappings/state" |
Contributor
There was a problem hiding this comment.
should this follow the same pattern as instance where the makefile just passes whats in $(tf_state)?eg:
tf_state = -backend-config="bucket=$(bucket_name)"
avshetty1980
requested changes
May 7, 2026
Contributor
avshetty1980
left a comment
avshetty1980
left a comment
There was a problem hiding this comment.
Nearly there, just a few comments
- Added a step to adopt existing event source mappings in the deploy-backend workflow. - Introduced validation for event source mappings in the quality-checks workflow. - Updated the Makefile to improve handling of Terraform variable scope. - Enhanced the adopt_event_source_mappings.sh script for better resource management and logging. - Revised README.md to clarify the event source mapping adoption process.
- Added installation step for ShellCheck in the quality-checks workflow to improve shell script linting. - Updated Makefile to enhance validation of the `has_sub_environment_scope` variable, ensuring it is explicitly set to true or false, and improved error handling for missing or invalid values.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated blue/green Lambda trigger handoff in deployment by adding pre-plan state adoption and pre-apply stale-trigger cleanup steps to the backend workflow.
Added manage_blue_green_event_source_mappings.sh to resolve live mapping UUIDs, re-import shared delta and id-sync event source mappings into the target Terraform workspace, and delete obsolete side-specific mappings.
This removes the manual release checklist steps (“Disable delta” and “Disable ID sync”), making releases faster and reducing risk of human error during blue/green cutovers.