Skip to content

feat(storage): support delete source objects on compose #34665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nidhiii-27 wants to merge 3 commits into
base: main
Choose a base branch
from
+51 −3
Open

feat(storage): support delete source objects on compose #34665

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d76f79d
feat(storage): support delete source objects on compose
nidhiii-27 Jun 24, 2026
939b8a8
Merge branch 'main' into compose-delete-source-ruby
nidhiii-27 Jun 25, 2026
6b11df4
fix(storage): resolve PR comments for delete_source_objects samples
nidhiii-27 Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions google-cloud-storage/samples/acceptance/files_test.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,52 @@ def mock_cipher.random_key
end

refute_nil bucket.file remote_file_name

@torreypayne torreypayne Jun 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: What do you think about asserting that the source files remain intact in this default test block above? I looked at the previous PR that adds in the delete_source_objects param and I'm wondering if we can add an extra check to validate the delete_source_objects: false value doesn't accidentally delete the source files.

@nidhiii-27 nidhiii-27 Jun 26, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

refute_nil bucket.file file_1_name
refute_nil bucket.file file_2_name
end

it "compose_file with delete_source_objects" do

@torreypayne torreypayne Jun 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: For resiliency against edge cases, it would be awesome to add a test to cover how the client behaves if a compose operation fails midway (ex. due to a transient API error). Otherwise LGTM!

@nidhiii-27 nidhiii-27 Jun 26, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Co-authored by AI Agent

file_1 = bucket.create_file local_file, file_1_name
file_2 = bucket.create_file local_file, file_2_name

expected_out = "Composed new file #{remote_file_name} in the bucket #{bucket.name} " \
"by combining #{file_1.name} and #{file_2.name}\n" \
"Source objects were deleted\n"
assert_output expected_out do
compose_file bucket_name: bucket.name,
first_file_name: file_1.name,
second_file_name: file_2.name,
destination_file_name: remote_file_name,
delete_source_objects: true
end

refute_nil bucket.file remote_file_name
assert_nil bucket.file file_1_name
assert_nil bucket.file file_2_name
end

it "compose_file with delete_source_objects failing does not delete source objects" do
file_1 = bucket.create_file local_file, file_1_name
file_2 = bucket.create_file local_file, file_2_name

real_storage = Google::Cloud::Storage.new
bucket.stub :compose, ->(*) { raise Google::Cloud::Error, "Mock API error" } do
real_storage.stub :bucket, bucket do
Google::Cloud::Storage.stub :new, real_storage do
assert_raises Google::Cloud::Error do
compose_file bucket_name: bucket.name,
first_file_name: file_1.name,
second_file_name: file_2.name,
destination_file_name: remote_file_name,
delete_source_objects: true
end
end
end
end

refute_nil bucket.file file_1_name
refute_nil bucket.file file_2_name
assert_nil bucket.file remote_file_name
end

it "copy_file" do
Expand Down
8 changes: 5 additions & 3 deletions google-cloud-storage/samples/storage_compose_file.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.

# [START storage_compose_file]
def compose_file bucket_name:, first_file_name:, second_file_name:, destination_file_name:
def compose_file bucket_name:, first_file_name:, second_file_name:, destination_file_name:, delete_source_objects: false
# The ID of your GCS bucket
# bucket_name = "your-unique-bucket-name"

Expand All @@ -31,18 +31,20 @@ def compose_file bucket_name:, first_file_name:, second_file_name:, destination_
storage = Google::Cloud::Storage.new
bucket = storage.bucket bucket_name, skip_lookup: true

destination = bucket.compose [first_file_name, second_file_name], destination_file_name do |f|
destination = bucket.compose [first_file_name, second_file_name], destination_file_name, delete_source_objects: delete_source_objects do |f|
f.content_type = "text/plain"
end

puts "Composed new file #{destination.name} in the bucket #{bucket_name} " \
"by combining #{first_file_name} and #{second_file_name}"
puts "Source objects were deleted" if delete_source_objects
end
# [END storage_compose_file]

if $PROGRAM_NAME == __FILE__
compose_file bucket_name: ARGV.shift,
first_file_name: ARGV.shift,
second_file_name: ARGV.shift,
destination_file_name: ARGV.shift
destination_file_name: ARGV.shift,
delete_source_objects: ARGV.shift == "true"
end
Loading