Skip to content

ateapi: backfill unset worker state during pod sync #599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mesut Oezdil (mesutoezdil) wants to merge 1 commit into
base: main
Choose a base branch
from
+4 −4
Open

ateapi: backfill unset worker state during pod sync #599

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions cmd/ateapi/internal/controlapi/syncer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ func (s *WorkerPoolSyncer) Start(ctx context.Context) {

// Reconcile the other direction: enqueue every stored worker so records
// whose pods no longer exist are cleaned up. This recovers delete events
// missed while ate-api-server was down — neither the watch relist nor
// missed while ate-api-server was down, neither the watch relist nor
// the resync period can replay a delete across a process restart,
// because the informer cache starts empty. Runs after the cache sync so
// the indexer is an authoritative snapshot of live pods.
Expand Down Expand Up @@ -221,7 +221,7 @@ func (s *WorkerPoolSyncer) reconcile(ctx context.Context, key workerKey) error {
if pod.DeletionTimestamp != nil {
// The pod has entered Terminating: mark the worker DRAINING so the
// scheduler stops routing new actors to it. We deliberately do NOT touch
// the bound actor here — inside the pod ateom has received SIGTERM and is
// the bound actor here, inside the pod ateom has received SIGTERM and is
// gracefully shutting the actor down. Actor cleanup happens on the Pod
// Deleted event.
return s.markWorkerDraining(ctx, key)
Expand Down Expand Up @@ -348,7 +348,7 @@ func workerCapacity(pod *corev1.Pod) *ateapipb.WorkerCapacity {

// markWorkerDraining transitions a worker to STATE_DRAINING so the scheduler
// stops routing new actors to it while its pod is Terminating. If the worker is
// already gone or already draining there is nothing more to do — the Pod
// already gone or already draining there is nothing more to do, the Pod
// Deleted event will clean up the record. A version conflict is returned so the
// caller requeues and retries against the updated record.
func (s *WorkerPoolSyncer) markWorkerDraining(ctx context.Context, key workerKey) error {
Expand Down
2 changes: 1 addition & 1 deletion cmd/ateapi/internal/controlapi/syncer_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ func setupReconcileTest(t *testing.T, persistence store.Interface, initPools ...

// TestSyncer_SoftDelete_MarksDraining verifies that a pod entering Terminating
// (DeletionTimestamp set) flips its worker to STATE_DRAINING without deleting the
// worker record or touching the bound actor — the actor is still gracefully
// worker record or touching the bound actor, the actor is still gracefully
// shutting down inside the pod.
func TestSyncer_SoftDelete_MarksDraining(t *testing.T) {
ctx := context.Background()
Expand Down