Dear dCache devs,
On our test server running today's 12.1.0.19b5372 master snapshot, I noticed some client delays. The situation:
- Accessing API with bearer token authentication, invalid token. Was testing how Ada handles this. I tried these cases:
- A cut off macaroon token
[invalid], with square brackets, as a token
- A completely empty token
- gPlazma is configured for a few OIDC providers
- One of the OIDC provides does not respond
In this situation, the client seems to hang; after 5 minutes, the reply is 500 (internal server error).
[onno@mylaptop ~] % time curl -H "Authorization: Bearer MDAxY2xvY2F0aW9uIE9wdGlvbmFsLmVtcHR5CjAwMThpZGVudGlmaWV" "https://dcachetest.grid.surfsara.nl:20443/api/v1/user" --fail
curl: (22) The requested URL returned error: 500
curl -H "https://dcachetest.grid.surfsara.nl:20443/api/v1/user" --fail 0,02s user 0,03s system 0% cpu 3:00,44 total
In the gPlazma log:
11 Aug 2026 11:44:10 (gPlazma) [Frontend-badger14 Login] Login attempt failed; detailed explanation follows:
LOGIN FAIL
| in: Origin[2001:610:450:b0::3:7]
| BearerTokenCredential[MDAx+{Hash=gAUJ1o24s3s}+maWV]
| out: Origin[2001:610:450:b0::3:7]
|
+--AUTH OK
| |
| +--x509 OPTIONAL:FAIL (no X.509 certificate chain) => OK
| |
| +--voms OPTIONAL:FAIL (no X509 certificate chain) => OK
| |
| +--kpwd OPTIONAL:FAIL (no username and password) => OK
| |
| +--jaas OPTIONAL:FAIL (no login name) => OK
| |
| +--ldap OPTIONAL:FAIL (no login name) => OK
| |
| +--oidc OPTIONAL:FAIL (Shutting down) => OK
|
+--MAP OK
| |
| +--vorolemap OPTIONAL:FAIL (no record) => OK
| |
| +--mutator OPTIONAL:OK => OK
| |
| +--authzdb SUFFICIENT:FAIL (no mappable principal) => OK
| |
| +--multimap OPTIONAL:FAIL (no mappable principals) => OK
| |
| +--multimap OPTIONAL:FAIL (no mappable principals) => OK
| |
| +--multimap SUFFICIENT:FAIL (no mappable principals) => OK
| |
| +--multimap SUFFICIENT:FAIL (no mappable principals) => OK
| |
| +--multimap SUFFICIENT:FAIL (no mappable principals) => OK
| |
| +--kpwd SUFFICIENT:FAIL (no mappable principals) => OK
| |
| +--ldap SUFFICIENT:FAIL (no username) => OK
|
+--ACCOUNT OK
| |
| +--banfile REQUISITE:OK => OK
| |
| +--kpwd SUFFICIENT:OK => OK (ends the phase)
|
+--SESSION OK
| |
| +--roles REQUISITE:OK => OK
| |
| +--authzdb SUFFICIENT:FAIL (no username principal) => OK
| |
| +--kpwd SUFFICIENT:FAIL (no record found) => OK
| |
| +--ldap SUFFICIENT:OK => OK (ends the phase)
|
+--VALIDATION FAIL (no username, no UID, no primary GID)
And then, a timeout for the OIDC provider:
11 Aug 2026 11:45:34 (gPlazma) [] Failed to fetch discovery document for SKA: org.apache.http.conn.ConnectTimeoutException: Connect to ska-iam.stfc.ac.uk:443 [ska-iam.stfc.ac.uk/130.246.217.41, ska-iam.stfc.ac.uk/2001:630:54:1e:82f6:d929:0:0] failed: Connection timed out
When I remove the OIDC provider that doesn't respond from the gPlazma config, the client receives an immediate response:
[onno@mylaptop ~] % time curl -H "Authorization: Bearer MDAxY2xvY2F0aW9uIE9wdGlvbmFsLmVtcHR5CjAwMThpZGVudGlmaWV" "https://dcachetest.grid.surfsara.nl:20443/api/v1/user" --fail
curl: (22) The requested URL returned error: 401
curl -H "https://dcachetest.grid.surfsara.nl:20443/api/v1/user" --fail 0,02s user 0,01s system 14% cpu 0,228 total
My conclusion is, that gPlazma, when it gets a clearly invalid token, does not invalidate it, but still tries to contact OIDC providers. If my understanding is correct, this may have a few downsides:
- It is unnecessary
- I haven't checked which information is being sent to the OIDC providers, but it may be too much, which may expose private information to parties that shouldn't get it
- It may take time & some (CPU and network) resources, slowing down gPlazma
- If OIDC providers are slow or don't respond at all, it leads to client timeouts
- Hanging clients may take up unnecessary resources like open connections on doors
My request: please improve token validation and if they're garbage or otherwise invalid, fail early and don't bother contacting OIDC providers.
Would love to hear what you guys think of it.
Thanks,
Onno
Dear dCache devs,
On our test server running today's
12.1.0.19b5372master snapshot, I noticed some client delays. The situation:[invalid], with square brackets, as a tokenIn this situation, the client seems to hang; after 5 minutes, the reply is 500 (internal server error).
In the gPlazma log:
And then, a timeout for the OIDC provider:
When I remove the OIDC provider that doesn't respond from the gPlazma config, the client receives an immediate response:
My conclusion is, that gPlazma, when it gets a clearly invalid token, does not invalidate it, but still tries to contact OIDC providers. If my understanding is correct, this may have a few downsides:
My request: please improve token validation and if they're garbage or otherwise invalid, fail early and don't bother contacting OIDC providers.
Would love to hear what you guys think of it.
Thanks,
Onno