Complete rewrite of database layer - #2
Open
TheTechRobo wants to merge 16 commits into
Open
Conversation
Systems other than brozzler will probably be implemented with a configuration option, not this enum. Actionless jobs (i.e. JobType.NONE) are a hack. If they are desired, they can be emulated with a tag that owned by no pipelines, which avoids tainting the entire system's design with them. (This could be useful for, e.g., a simple way of storing results from custom JS over multiple pages, where otherwise you would need to merge the CUSTOM_JS result for each page.)
This makes the system (at least IMO) much easier to understand. Having one set of rules that could change four different settings took a lot more thinking than needed.
…rs from URLs before queuing.
Only searches through pages, not requisites. Also clean up the existing page listing code into nicer templates.
This also makes update_job_status only required to be called when JobExhausted is raised; completing an attempt computes a new status automatically. Also removes claim locks, I can't see a use for them and don't want them cluttering up the code any more.
Previously, the trigger assumed that (a) all discovered children would have a reduced depth, and (b) the difference in depth was the same for each one. It was also rather difficult for me to reason about months after writing it. AFAICT it's the only good way of doing it with a recursive CTE, so we're recursing manually now :-) I discovered this when playing around with adding a fuzz tester for the relations system. Two previously-failing tests have been added (one of them a fuzz test, the other a simplified case from that test) and one test that did pass but is good to have anyway. I did not realise how poorly-tested the relations system was until now. The tests should probably be cleaned up at some point, but I've spent too long looking at them these past few days, so that'll have to wait :P
Also add a 'script' service for docker-compose, for convenience.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First commit was from February, geez...
things still to do (off the top of my head):