Skip to content

Delegate AWS endpoint environment handling #569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
srinivasBJ wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Delegate AWS endpoint environment handling #569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions HISTORY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## UNRELEASED

- Delegated `AWS_ENDPOINT_URL` handling to boto3 now that the AWS SDK supports endpoint environment variables directly (Issue [#194](https://github.com/drivendataorg/cloudpathlib/issues/194)).
- Fixed mypy 2.x type errors in `Client` and `CloudPath` that caused CI lint failures (Issue [#563](https://github.com/drivendataorg/cloudpathlib/issues/563), PR [#566](https://github.com/drivendataorg/cloudpathlib/pull/566))
- Changed `S3Client._get_metadata` to read object metadata with `HeadObject` instead of `GetObject`, so `stat`, `etag`, and `size` no longer open the object body. Also fixes a `KeyError` on `ContentLength` against S3-compatible gateways that drop `Content-Length` from `GetObject` responses. (Issue [#564](https://github.com/drivendataorg/cloudpathlib/issues/564), PR [#565](https://github.com/drivendataorg/cloudpathlib/pull/565))

Expand Down
1 change: 0 additions & 1 deletion cloudpathlib/s3/s3client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ def __init__(
args that are supported look at the upload and download lists in the
[boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer).
"""
endpoint_url = endpoint_url or os.getenv("AWS_ENDPOINT_URL")
if boto3_session is not None:
self.sess = boto3_session
else:
Expand Down
9 changes: 5 additions & 4 deletions tests/test_s3_specific.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,12 +254,10 @@ def test_extra_args_via_requester_pays(s3_rig, tmp_path):


def test_aws_endpoint_url_env(monkeypatch):
"""Allows setting endpoint_url from env variable
until upstream boto3 PR is merged.
https://github.com/boto/boto3/pull/2746
"""
"""Delegates AWS_ENDPOINT_URL handling to boto3."""
s3_url = "https://s3.amazonaws.com"
localstack_url = "http://localhost:4566"
explicit_url = "http://localhost:9000"

s3_client = S3Client()
assert s3_client.client.meta.endpoint_url == s3_url
Expand All @@ -268,6 +266,9 @@ def test_aws_endpoint_url_env(monkeypatch):
s3_client_custom_endpoint = S3Client()
assert s3_client_custom_endpoint.client.meta.endpoint_url == localstack_url

s3_client_explicit_endpoint = S3Client(endpoint_url=explicit_url)
assert s3_client_explicit_endpoint.client.meta.endpoint_url == explicit_url


def test_as_url_local(monkeypatch):
path = S3Path("s3://arxiv/pdf")
Expand Down