Imaging Data Commons S3 bucket integration is broken

[fedorov]

Trying to open this URL built by IDC Portal https://volview.kitware.app/?urls=%5Bs3%3A%2F%2Fidc-open-data-two%2Ffe9c02f5-1cbb-4c3b-93a0-84b03fc94595%5D, I am getting these errors.

The issue does not appear to be on the IDC side, since the bucket content can be downloaded, without authentication:

$ pip install s5cmd
$ s5cmd --no-sign-request --endpoint-url https://s3.amazonaws.com cp "s3://idc-open-data-two/fe9c02f5-1cbb-4c3b-93a0-84b03fc94595/*" .

[PaulHax]

Thanks @fedorov, reproduced. VolView is hitting a CORS preflight rejection on the bucket listing network call.

VolView uses @aws-sdk/client-s3, which auto-attaches these telemetry/retry headers:

• x-amz-content-sha256
• x-amz-user-agent
• amz-sdk-invocation-id
• amz-sdk-request

Any one of them forces a preflight, and the bucket's preflight response includes Access-Control-Allow-Methods: GET, HEAD and Access-Control-Allow-Origin: * but no Access-Control-Allow-Headers — so the SDK headers aren't whitelisted and the request is blocked.

For an immidiate fix, the S3 bucket could add the headers the @aws-sdk/client-s3 includes.

 {
   "AllowedOrigins": ["*"],
   "AllowedMethods": ["GET", "HEAD"],
   "AllowedHeaders": [
     "Range",
     "amz-sdk-invocation-id",
     "amz-sdk-request",
     "x-amz-content-sha256",
     "x-amz-user-agent"
   ],
   "ExposeHeaders": ["Content-Range", "Content-Length", "Accept-Ranges"]
 }

Dropping the aws s3 library here to be more robust againts public buckets. #871

[fedorov]

@PaulHax thank you for such a prompt response!

I think it is better we wait until you remove this dependency. Updating IDC S3 CORS is not something that is super straightforward to do, and in general I would prefer not to add such granular headers to the configuration.