Merged
Conversation
Merge main into 359
Merge main into 359
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR moves ETL error handling and aggregation logic from HydroServer into hydroserverpy.
See hydroserver2/hydroserverpy#25
HydroServer's Celery tasks now use the ETLPipeline from hydroserverpy to build and run ETL tasks. All user-facing errors should be of type ETLError. Error messages have been redefined at the source of each exception rather than at an intermediate layer, so many messages have changed from their previous versions — we can adjust them as needed.
The frontend has not been updated yet, although the PR makes very few changes to the API. The structure of task run results has changed to include more context, and some fields such as the source URI have moved within the response — these will need to be accounted for on the frontend. We may want to review how tasks are stored in HydroServer to make sure we're using a design that makes sense across ETL, aggregation, QA/QC, and any other types of tasks we anticipate supporting. In the meantime, any mapping of configurations stored in HydroServer to the new ETL package interface is handled by a small helper module in hydroserverpy at https://github.com/hydroserver2/hydroserverpy/blob/359-etl-error-handling/src/hydroserverpy/etl/hydroserver.py
This updated ETL package is currently deployed to https://hydroserverdev.waterrights.utah.gov/