Scheduled functions: Concurrent execution by not awaiting on current run. - #5736
Open
Shubham8287 wants to merge 3 commits into
Open
Scheduled functions: Concurrent execution by not awaiting on current run.#5736Shubham8287 wants to merge 3 commits into
awaiting on current run.#5736Shubham8287 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
handle_queuedonly finishes after function associated with current item returns. It causes head-of-line blocking and cause scheduled reducers to get delayed behind slow procedures.My intention was to fix only reducer behind procedure case but it turns out a general solution is easier to implement.
handle_queuenow do not wait for function to finish but only adds a future inScheduledActor::active_calls, which we poll later inScheduledActor::run. Concurrency is bounded atModuleHostlevel similar to normal procedures /reducersSome implicit behavior changes:
Scheduled functions are drained from
SchedulerActor’s delay queue earlier and submitted to the databaseexecutorqueue sooner.SchedulerActor::closemay now take longer, because it waits for all already-dispatched scheduled callsin db
executor, not just the single call that was previouslyawaited inline.(Major one) For already-expired scheduled functions, execution order is not guaranteed to match
scheduled_atorder. For example, if items scheduled for T- 5ms, T are all expired by the time the scheduler
observes them, there is no guarantee that the T-5ms item is submitted before the T item though time observed from inside the reducer will still be monotonic.
API and ABI breaking changes
NA
Expected complexity level and risk
3, there could be unexpected implications related to module cleanup, resource exhaustion, etc.
Testing
Old tests has been modified for new behaviour.