Exclude noisy subgraph errors from being logged as part of "unhandled exception" logging

[dnechay]

Is your feature request related to a problem? Please describe.
In web services we have a global exception filter that logs "unhandled exception". When subgraph is laggy and throws "bad indexers" error - such errors are being logged as unhandled and hence we get alerted. We can try specifying some indexer that is more stable, but it can lead to the same situation, so better would be to just ignore such errors from being logged, so we aren't alerted with this noisy error.

Errors we noticed so far:

• bad indexers
• 521
• 502
• just a fetch failed from underlying undici client in graph sdk

Describe the solution you'd like
Ideally we should have some custom error class in SDK and throw it in case subgraph request fails (e.g. SubgraphRequestFailedError), then in apps we will be able to ignore the log and return 502 just by checking error instance type.

Describe alternatives you've considered
Considered handling errors on apps directly, but that's code duplication and breaks boundaries.

Additional context

• you can check Datadog logs e.g. for HUMAN Dashboard in order to get error examples
• it should be implemented in all JS-based web services

[portuu3]

@dnechay how will we then realize of the errors happening?

[dnechay]

@portuu3 unhandled exceptions is some very unusual situation that should not happen often and usually indicate about bugs, but what we have with subgraph is kinda known to us and we don't need it as part of unhandled logging. Instead, if SDK can throw some meaningful error that we able to distinguish from others, then we can:

1. Exclude such errors from being logger as "unhandled"
2. Have another log for "subgraph errors", e.g. warn level or error with custom message
3. In order to get alerted if something weird is happening on subgraph - we either set up an alert on the specific error log from point 2, or can setup alert on number of 5xx, because even if we skip such errors from logging (or have them as debug that is logger only when enabled by env) - status code returned still might be e.g. 502/503 and we can monitor+alert on it

But my main ideas was to remove it from "unhandled" logging to avoid noisy alerts that we have now because intermittent subgraph errors (1 or 2 in a day) and create general monitors on 5xx to alert about any unreliability, not just unhandled exceptions

cc @flopez7 @Dzeranov

[flopez7]

Agree that it makes sense to exclude “bad indexer” errors since they come from subgraph we don't have to do anything to resolve them. But for other errors like 502, 504, or general fetch failures, I think we should still log them as unhandled. That way we keep visibility and get notified when something more serious goes wrong, otherwise we might miss real issues @dnechay.
cc: @portuu3 @Dzeranov