Skip to content

Fix task log persistence on Postgres and harden the log flush path#212

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix-task-log-db-persistence
Open

Fix task log persistence on Postgres and harden the log flush path#212
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix-task-log-db-persistence

Conversation

@damilolaedwards

Copy link
Copy Markdown

Problem

Two related problems in the task log persistence path, both specific to the write to the database. SQLite, the default engine, was not affected by the first one.

Task logs are never stored on Postgres

InsertTaskLog upserts with ON CONFLICT (run_id, task_id, log_time), but the only unique key on task_logs is the primary key (run_id, task_id, log_idx); there is no unique index on log_time. Postgres rejects an ON CONFLICT target that does not match a unique or exclusion constraint, so every task log insert fails on Postgres at execution time and no task logs are ever stored. The SQLite branch uses INSERT OR REPLACE keyed by the primary key and works.

The flush path fails badly on any flush error

Whenever a flush fails (on Postgres, every time), three things went wrong:

  • The failure was logged through the writer's own logger, whose database hook re-enters Fire and tries to take the buffer mutex that the flush already holds, so the logging goroutine deadlocks.
  • The buffer was only cleared on success, so on a persistent failure it grows without bound.
  • The flush coordination used a shared cancellation channel that could be closed twice and panic.

Fix

  • Point the upsert conflict target at the primary key (run_id, task_id, log_idx), which both engines share.
  • Rewrite the flush coordination around the buffer mutex and a single scheduled flag. The buffer is always cleared after a flush attempt, so a failing database cannot grow it without bound. Flush errors are reported through the parent logger instead of the writer's own logger, so they cannot re-enter the hook. The cancellation channel is removed: a delayed flush simply finds an empty buffer if an earlier flush already ran, which also removes the double close.

Tests

  • Database layer: a task log is persisted and re-inserting the same primary key updates the row in place rather than duplicating or failing.
  • Logger: against a database whose table does not exist so every flush fails, logging keeps making progress and the buffer stays bounded; and on a working database, entries logged through the scope are persisted on flush.

go build ./..., go vet, and the logger and db packages under -race pass.

The task log upsert used ON CONFLICT (run_id, task_id, log_time), but the
only unique key on task_logs is the primary key (run_id, task_id,
log_idx). Postgres rejects an ON CONFLICT target that does not match a
unique constraint, so every task log insert failed on Postgres and no
task logs were stored. Point the conflict target at the primary key,
which both engines share.

The flush path also failed badly whenever a flush errored, which on
Postgres was every time. On error it logged through the writer's own
logger, whose database hook re-entered the flush and deadlocked on the
buffer mutex; the buffer was only cleared on success, so it grew without
bound; and the flush used a shared channel that could be closed twice and
panic.

Rewrite the flush coordination around the buffer mutex and a scheduled
flag. The buffer is always cleared after a flush attempt, flush errors are
reported through the parent logger so they cannot re-enter the hook, and
the cancellation channel is removed, so a delayed flush simply finds an
empty buffer when an earlier flush already ran. Add tests for persistence
and upsert, for progress and a bounded buffer under a failing database,
and for the normal persistence path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant