Skip to content

revert: remove open-world evidence foundation - #413

Merged
yyiilluu merged 4 commits into
mainfrom
codex/revert-pr-407
Aug 4, 2026
Merged

revert: remove open-world evidence foundation#413
yyiilluu merged 4 commits into
mainfrom
codex/revert-pr-407

Conversation

@yyiilluu

@yyiilluu yyiilluu commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Changes

Evidence foundation rollback

  • Remove search-exposure recording and session-outcome identity helpers.
  • Restore the prior session outcome schemas, client surface, and SQLite/storage contracts.
  • Restore the prior governance erase/claim flow and retention behavior.
  • Restore the prior resumable extraction and learning-billing behavior.

Dependent follow-ups

Review follow-ups

  • Rebuild feat: add open-world evidence foundation #407-era SQLite session_outcomes tables into the restored schema, preserving success/failure rows, backfilling governance subject references, and explicitly dropping unrepresentable unknown outcomes with a warning.
  • Restore the SQLite 3.35 minimum required by existing RETURNING and DROP COLUMN usage.
  • Make outcome erasure resilient to governance-secret rotation and return a stable session_outcomes deletion count.
  • Acquire SQLite governance write locks before state checks, serialize idempotent purge begin/prepare flows across connections, and roll back failed target writes so SQLite cannot retain a stale writer transaction.\n- Reject legacy session-outcome schemas with empty governance-subject defaults and rebuild them with derived subject references.\n- Make synchronous playbook/profile persistence atomic while keeping scheduler dispatch strictly post-commit.
  • Meter resumable extraction from persisted survivors only, use retry-stable fallback keys, and emit learning billing from incremental aggregation.
  • Treat post-persist optimization and aggregation scheduling failures as best-effort side effects.

Validation follow-up

  • Represent the callback executor's last anomaly time with an explicit unset sentinel so the first threshold crossing is never suppressed by low system uptime.

Test Plan

  • uv run ruff check reflexio tests
  • uv run ruff format --check reflexio tests
  • Pyright on all 23 staged Python files: 0 errors, 0 warnings
  • Latest affected review files: 301 passed
  • OSS non-E2E suite: 5,535 passed, 73 skipped, 6 subtests passed
  • OSS E2E suite: 47 passed, 51 skipped
  • npm --prefix docs run lint: 0 errors (3 existing warnings)
  • cd docs && npx tsc --noEmit
  • python -c "import reflexio"

Reverts 85a4b2255a96ef2a5b50f4cbe7c10758439e76b3, plus dependent follow-ups 785a9e053ff771f40704bb7b0b5bbbe36048806a and eb88f44fd3b53457b76e8500ac1a30ba7d4ab16e.

Summary by CodeRabbit

  • Changes

    • Session outcomes now support only success or failure, with simpler responses and retry behavior.
    • Governance data erasure workflows have streamlined retry and completion handling, including session-outcome removal.
    • Search exposure event recording has been removed; search results and metering remain available.
    • Learning-generation billing supports durable per-record tracking, retry-stable keys, and count-based fallback.
    • Scheduler failures during playbook processing are logged without preventing other scheduled actions.
  • Documentation

    • Quick Start prerequisites now list Node.js without the previous SQLite verification step.
    • Billing and extraction guidance has been updated.

Revert squash commit 85a4b22 (#407) and its dependent follow-ups 785a9e0 (#408) and eb88f44 (#409). Preserve the independent aggregation work that landed afterward.
Use an explicit unset sentinel for the last anomaly time so hosts with less than one hour of monotonic uptime do not suppress the first threshold crossing.
@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The change simplifies session outcomes, removes finalization receipts and purge execution claims, updates learning billing to use durable IDs or counts, removes selected retention and search-exposure infrastructure, and revises related storage, documentation, and tests.

Changes

Session outcome simplification

Layer / File(s) Summary
Outcome contracts and storage
docs/lib/methods/requests-sessions.ts, reflexio/lib/_session_outcome.py, reflexio/models/api_schema/domain/*, reflexio/server/services/storage/*
Session outcomes now support only success and failure. Identity fields, digests, related validation, and legacy migrations were removed.
Outcome tests
tests/server/api_endpoints/*, tests/server/services/storage/*
Tests now validate core outcome fields and reduced responses without identity or conflict cases.

Learning finalization and billing

Layer / File(s) Summary
Finalization flow
reflexio/server/services/{base_generation_service.py,deferred_learning_plan.py}, reflexio/server/services/{profile,playbook}/*, reflexio/server/services/extraction/resume_worker.py
Finalization no longer returns receipt results or learning IDs.
Billing emission
reflexio/server/billing_meter.py, reflexio/server/services/playbook/components/aggregator.py, tests/server/services/test_non_extraction_learning_metering.py
Billing emits entity-backed events when durable IDs are complete and count-based events otherwise.

Governance purge simplification

Layer / File(s) Summary
Purge and barrier APIs
reflexio/server/services/governance/*, reflexio/server/services/storage/storage_base/governance/*
Execution-claim and authoritative-user parameters and lifecycle methods were removed.
SQLite implementation and tests
reflexio/server/services/storage/sqlite_storage/governance/*, tests/server/services/governance/*, tests/server/services/storage/sqlite_storage/test_governance_storage.py
SQLite governance operations use direct purge and barrier calls, subject references, and idempotent completion.

Retention and runtime cleanup

Layer / File(s) Summary
Retention and setup behavior
README.md, reflexio/server/services/storage/retention.py, reflexio/server/services/storage/retention_mixin.py, reflexio/server/callback_executor.py
The SQLite prerequisite check and retention age filtering were removed. Selected retention targets were deleted. Anomaly throttling now treats an unset timestamp explicitly.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Suggested reviewers: guangyu-reflexio, yilu331

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 18.70% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the primary change: reverting the open-world evidence foundation and its dependent changes.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/revert-pr-407

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (4)
reflexio/server/services/storage/sqlite_storage/_base.py (2)

2468-2480: 🗄️ Data Integrity & Integration | 🔴 Critical | 🏗️ Heavy lift

Add a guarded rebuild migration for session_outcomes.

CREATE TABLE IF NOT EXISTS leaves the #407 table unchanged. Its NOT NULL columns include outcome_id, outcome_revision, outcome_contract_digest, and finalized_trajectory_digest. The current insert omits these columns and fails with NOT NULL constraint failed: session_outcomes.outcome_id. Add the migration to migrate(), preserve compatible rows, handle existing 'unknown' rows, and recreate the indexes.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reflexio/server/services/storage/sqlite_storage/_base.py` around lines 2468 -
2480, The session_outcomes schema needs a guarded rebuild migration because the
existing `#407` table has additional NOT NULL columns that the current inserts
omit. Add migration logic in migrate() to detect the incompatible schema, create
the corrected table including outcome_id, outcome_revision,
outcome_contract_digest, and finalized_trajectory_digest, preserve compatible
rows while mapping existing 'unknown' outcomes appropriately, swap the tables
safely, and recreate all session_outcomes indexes.

666-667: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Keep the SQLite 3.35.0 guard or provide compatible fallbacks.

DROP COLUMN and RETURNING both require SQLite 3.35.0 or newer. Older SQLite runtimes can fail during migration or later storage operations.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reflexio/server/services/storage/sqlite_storage/_base.py` around lines 666 -
667, Preserve the SQLite 3.35.0 version guard in the storage initialization and
migration flow around the database path setup. For older runtimes, provide
compatible fallbacks for every DROP COLUMN and RETURNING operation, ensuring
migrations and subsequent storage operations continue without relying on
unsupported SQLite syntax.
reflexio/server/services/extraction/resume_worker.py (1)

887-905: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Billing counts pre-persist items, not persisted rows.

items is the list handed to _finalize_extracted_items. That method runs _resolve_write_plan, which consolidates and can drop candidates before persist. _record_finalized_learnings then meters items, so dropped candidates are still billed:

  • Playbook: a dropped item keeps user_playbook_id=0, the code falls back to count=len(items), and the dropped item is counted.
  • Profile: profile_id is assigned by the extractor before persist, so a consolidated-away profile still produces an entity-backed event for a row that was never written.

Meter the persisted rows instead. _finalize_extracted_items already builds the write plan, so return or expose the persisted survivors and pass those to _record_finalized_learnings.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reflexio/server/services/extraction/resume_worker.py` around lines 887 - 905,
Update _finalize_extracted_items and both profile/playbook callers in the resume
worker so finalization returns or otherwise exposes the persisted survivor rows
produced by _resolve_write_plan. Pass only those survivors to
_record_finalized_learnings, preserving the existing entity_type values, rather
than passing the original items list.
reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py (1)

375-384: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Use BEGIN IMMEDIATE for the validate-then-delete transaction.

With WAL enabled, a competing writer can commit after validation. The deferred transaction then fails with SQLITE_BUSY when its snapshot is upgraded for deletion. Use BEGIN IMMEDIATE to acquire the write lock before validation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py`
around lines 375 - 384, Update the transaction start in the validate-then-delete
flow containing _validate_prepared_delete_target_matrix_locked and
_validate_hide_for_rebuild_targets_locked to use BEGIN IMMEDIATE instead of
BEGIN, acquiring the write lock before validation while leaving the subsequent
validation and deletion steps unchanged.
🧹 Nitpick comments (1)
reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py (1)

271-276: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Return a stable key set from _clear_user_data_for_governance_locked.

The session_outcomes key now appears only when rowcount is truthy. Every other key is always present. The method declares dict[str, int], and apply_governance_user_data_delete returns this dict to callers. A caller that indexes counts["session_outcomes"] raises KeyError for the common zero-deletion case, while clear_session_outcomes_for_user in reflexio/server/services/storage/sqlite_storage/_session_outcomes.py always returns that key.

apply_governance_user_data_delete already reads the value with counts.get(key, 0) on line 392, so always emitting the key changes nothing in the delete-target matrix. It does require updating the exact-equality assertion in tests/server/services/storage/sqlite_storage/test_governance_storage.py around line 3273.

♻️ Proposed refactor
         return {
-            **(
-                {"session_outcomes": session_outcomes_cur.rowcount}
-                if session_outcomes_cur.rowcount
-                else {}
-            ),
+            "session_outcomes": session_outcomes_cur.rowcount,
             "interactions": interactions_cur.rowcount,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py`
around lines 271 - 276, Update _clear_user_data_for_governance_locked to always
include the session_outcomes key, using session_outcomes_cur.rowcount even when
it is zero, while preserving the existing values for all other keys. Update the
exact expected dictionary assertion in test_governance_storage.py to include
session_outcomes: 0 for the zero-deletion case.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@reflexio/models/api_schema/domain/entities.py`:
- Around line 957-958: Update the comment describing the clear-user-data scope
near the relevant clear-user-data declaration to explicitly include session
outcomes alongside interactions, requests, playbooks, and profiles; do not
change the deletion behavior.

In `@reflexio/server/billing_meter.py`:
- Around line 93-97: The count fallback in the finalized-learning recording flow
must reuse a stable idempotency key across retries instead of generating a new
UUID per call. Update the fallback around _record_finalized_learnings and its
billing-meter helper to derive the event key from the finalized run and batch,
or persist and reuse an idempotency marker before emitting the aggregate event;
retain the existing no-op behavior for count <= 0.

In `@reflexio/server/services/extraction/resume_worker.py`:
- Around line 318-321: Update record_learnings_generated in the finalization
flow to use a deterministic batch event key derived from run.id instead of
generating a new UUID on each attempt. Ensure _retry_finalization reuses the
same key as the initial finalization so fallback learnings are not
double-counted.

In `@reflexio/server/services/playbook/components/aggregator.py`:
- Around line 1964-1974: Update _run_incremental to call
_record_learnings_generated before its return, using the saved playbooks’
durable agent_playbook_id values and the same total-count fallback behavior as
the existing aggregation path; add a regression test covering both durable IDs
and the fallback count.

In `@reflexio/server/services/playbook/service.py`:
- Around line 604-607: Update emit_generation_side_effects and
_finalize_extracted_items to contain exceptions from both
_enqueue_user_playbook_optimization and _trigger_playbook_aggregation locally
after persistence completes. Log each scheduler failure without re-raising it,
so post-commit finalization remains successful and retries cannot repeat durable
writes or billing.

In `@reflexio/server/services/storage/sqlite_storage/_governance.py`:
- Around line 407-411: Update the governance schema migration, including
_ensure_governance_subject_ref_columns, to cover session_outcomes: backfill
nullable or missing governance_subject_ref values for legacy rows using the
appropriate subject mapping, then enforce the existing NOT NULL constraint.
Ensure purge-completion logic cannot be bypassed by legacy session_outcomes
rows.

In `@reflexio/server/services/storage/sqlite_storage/_session_outcomes.py`:
- Around line 204-213: Update clear_session_outcomes_for_user to delete
session_outcomes by user_id as well as governance_subject_ref, so erasure still
removes rows created before secret rotation. Preserve the existing locking,
commit, and rowcount behavior while ensuring both current and previously derived
identifiers are covered.

In `@reflexio/server/services/storage/sqlite_storage/governance/_purge.py`:
- Around line 346-377: Update fail_purge_operation to roll back the connection
before either ValueError raised after a zero-row UPDATE, matching the try/except
rollback handling in fail_subject_erasure_barrier. Ensure the rollback occurs
before checking the existing operation status or raising the not-found error,
while preserving the existing successful commit and return behavior.

In `@reflexio/server/services/storage/storage_base/__init__.py`:
- Around line 150-151: Update the clear_user_data docstring to include session
outcomes among the deleted user data and explicitly state that session-outcome
deletions are intentionally omitted from the returned dictionary, while
preserving the existing description of agent_playbooks.

---

Outside diff comments:
In `@reflexio/server/services/extraction/resume_worker.py`:
- Around line 887-905: Update _finalize_extracted_items and both
profile/playbook callers in the resume worker so finalization returns or
otherwise exposes the persisted survivor rows produced by _resolve_write_plan.
Pass only those survivors to _record_finalized_learnings, preserving the
existing entity_type values, rather than passing the original items list.

In `@reflexio/server/services/storage/sqlite_storage/_base.py`:
- Around line 2468-2480: The session_outcomes schema needs a guarded rebuild
migration because the existing `#407` table has additional NOT NULL columns that
the current inserts omit. Add migration logic in migrate() to detect the
incompatible schema, create the corrected table including outcome_id,
outcome_revision, outcome_contract_digest, and finalized_trajectory_digest,
preserve compatible rows while mapping existing 'unknown' outcomes
appropriately, swap the tables safely, and recreate all session_outcomes
indexes.
- Around line 666-667: Preserve the SQLite 3.35.0 version guard in the storage
initialization and migration flow around the database path setup. For older
runtimes, provide compatible fallbacks for every DROP COLUMN and RETURNING
operation, ensuring migrations and subsequent storage operations continue
without relying on unsupported SQLite syntax.

In
`@reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py`:
- Around line 375-384: Update the transaction start in the validate-then-delete
flow containing _validate_prepared_delete_target_matrix_locked and
_validate_hide_for_rebuild_targets_locked to use BEGIN IMMEDIATE instead of
BEGIN, acquiring the write lock before validation while leaving the subsequent
validation and deletion steps unchanged.

---

Nitpick comments:
In
`@reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py`:
- Around line 271-276: Update _clear_user_data_for_governance_locked to always
include the session_outcomes key, using session_outcomes_cur.rowcount even when
it is zero, while preserving the existing values for all other keys. Update the
exact expected dictionary assertion in test_governance_storage.py to include
session_outcomes: 0 for the zero-deletion case.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 73256250-5714-46ee-a81f-6b7abcb8513d

📥 Commits

Reviewing files that changed from the base of the PR and between 159d8ab and f83d4fe.

📒 Files selected for processing (58)
  • README.md
  • docs/lib/methods/requests-sessions.ts
  • reflexio/client/client.py
  • reflexio/lib/_session_outcome.py
  • reflexio/models/api_schema/domain/entities.py
  • reflexio/models/api_schema/domain/enums.py
  • reflexio/server/billing_meter.py
  • reflexio/server/callback_executor.py
  • reflexio/server/routes/search.py
  • reflexio/server/services/base_generation/_usage_billing.py
  • reflexio/server/services/base_generation_service.py
  • reflexio/server/services/deferred_learning_plan.py
  • reflexio/server/services/extraction/README.md
  • reflexio/server/services/extraction/resume_worker.py
  • reflexio/server/services/governance/service.py
  • reflexio/server/services/playbook/components/aggregator.py
  • reflexio/server/services/playbook/service.py
  • reflexio/server/services/profile/service.py
  • reflexio/server/services/search_exposure.py
  • reflexio/server/services/storage/governance_claims.py
  • reflexio/server/services/storage/governance_validation.py
  • reflexio/server/services/storage/retention.py
  • reflexio/server/services/storage/retention_mixin.py
  • reflexio/server/services/storage/session_outcome_identity.py
  • reflexio/server/services/storage/sqlite_storage/_base.py
  • reflexio/server/services/storage/sqlite_storage/_governance.py
  • reflexio/server/services/storage/sqlite_storage/_session_outcomes.py
  • reflexio/server/services/storage/sqlite_storage/agent_run/_agent_run_store.py
  • reflexio/server/services/storage/sqlite_storage/base/_deletion.py
  • reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py
  • reflexio/server/services/storage/sqlite_storage/governance/_purge.py
  • reflexio/server/services/storage/sqlite_storage/governance/_subject_barrier.py
  • reflexio/server/services/storage/storage_base/__init__.py
  • reflexio/server/services/storage/storage_base/_session_outcomes.py
  • reflexio/server/services/storage/storage_base/agent_run/_agent_run_store.py
  • reflexio/server/services/storage/storage_base/governance/_erase_execution.py
  • reflexio/server/services/storage/storage_base/governance/_purge.py
  • reflexio/server/services/storage/storage_base/governance/_subject_barrier.py
  • tests/client/test_session_outcomes_client.py
  • tests/models/test_session_outcome_identity.py
  • tests/server/api_endpoints/test_session_outcomes_integration.py
  • tests/server/routes/test_search_exposure_boundary.py
  • tests/server/services/extraction/test_resume_worker.py
  • tests/server/services/governance/test_governance_local_e2e.py
  • tests/server/services/governance/test_subject_write_barrier_sqlite.py
  • tests/server/services/storage/sqlite_storage/test_agent_run_storage.py
  • tests/server/services/storage/sqlite_storage/test_governance_retrieved_learning.py
  • tests/server/services/storage/sqlite_storage/test_governance_storage.py
  • tests/server/services/storage/sqlite_storage/test_session_id_migration.py
  • tests/server/services/storage/sqlite_storage/test_session_outcome_migration.py
  • tests/server/services/storage/test_sqlite_storage.py
  • tests/server/services/storage/test_storage_contract_clear_user_data.py
  • tests/server/services/storage/test_storage_contract_retention.py
  • tests/server/services/storage/test_storage_contract_session_outcomes.py
  • tests/server/services/test_generation_billing_emission.py
  • tests/server/services/test_non_extraction_learning_metering.py
  • tests/server/services/test_search_exposure.py
  • tests/server/test_billing_meter_events.py
💤 Files with no reviewable changes (21)
  • README.md
  • tests/models/test_session_outcome_identity.py
  • tests/client/test_session_outcomes_client.py
  • reflexio/models/api_schema/domain/enums.py
  • reflexio/server/services/storage/session_outcome_identity.py
  • reflexio/server/services/storage/governance_validation.py
  • reflexio/server/services/storage/sqlite_storage/agent_run/_agent_run_store.py
  • reflexio/server/services/storage/storage_base/_session_outcomes.py
  • reflexio/server/services/storage/governance_claims.py
  • tests/server/services/storage/sqlite_storage/test_agent_run_storage.py
  • reflexio/server/services/search_exposure.py
  • tests/server/services/storage/test_storage_contract_retention.py
  • reflexio/server/routes/search.py
  • reflexio/server/services/storage/storage_base/agent_run/_agent_run_store.py
  • tests/server/services/storage/test_storage_contract_session_outcomes.py
  • tests/server/routes/test_search_exposure_boundary.py
  • tests/server/services/test_search_exposure.py
  • reflexio/server/services/storage/retention.py
  • tests/server/services/storage/test_sqlite_storage.py
  • tests/server/services/storage/test_storage_contract_clear_user_data.py
  • tests/server/services/storage/sqlite_storage/test_session_outcome_migration.py

Comment thread reflexio/models/api_schema/domain/entities.py Outdated
Comment thread reflexio/server/billing_meter.py
Comment thread reflexio/server/services/extraction/resume_worker.py
Comment thread reflexio/server/services/playbook/components/aggregator.py
Comment thread reflexio/server/services/playbook/service.py Outdated
Comment thread reflexio/server/services/storage/sqlite_storage/_governance.py
Comment thread reflexio/server/services/storage/storage_base/__init__.py Outdated
Preserve session-outcome compatibility across the #407 downgrade, make governance erasure and transactions robust, and keep learning metering retry-safe and aligned with persisted survivors.
@yyiilluu

yyiilluu commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

CodeRabbit review-body findings are addressed in 3c054d5d3a0adf00f274bf877b174b565e3d4f84:

  • SQLite downgrade compatibility — fixed. A transactional startup migration now rebuilds the feat: add open-world evidence foundation #407 identity schema into the restored schema, preserves success/failure rows, backfills missing governance refs, recreates indexes through the canonical DDL, and warns while dropping unknown rows that the reverted enum cannot represent.
  • SQLite minimum version — fixed. The 3.35 guard is restored because current storage code independently uses RETURNING and DROP COLUMN.
  • Persisted-survivor metering — fixed. Profile and playbook finalizers return their write-plan survivors, and resumable extraction bills only those rows.
  • Governance erase transaction race — fixed. apply_governance_user_data_delete now takes BEGIN IMMEDIATE before validation reads.
  • Stable deletion-count shape — fixed. session_outcomes is always present in clear/governance deletion counts, including zero.
  • Docstring coverage warning — no bulk change. This is a broad coverage metric rather than a concrete correctness defect. The PR is primarily a revert, and the newly introduced downgrade migration and changed public contracts are documented; adding unrelated docstrings across restored legacy code would expand the rollback beyond the reviewed behavior.

Validation after the fixes: affected files 341 passed; full non-E2E tier 4,859 passed, 11 skipped, 6 subtests; E2E 47 passed, 51 skipped; repository-wide Ruff clean; Pyright on all 23 staged Python files clean.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
reflexio/server/services/storage/sqlite_storage/governance/_purge.py (2)

260-271: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Roll back failed record_purge_target writes.

If either DML statement or commit() raises, call self.conn.rollback() before re-raising. Otherwise, the implicit transaction can remain active and hold SQLite write locks. Add a failure-path test that asserts storage.conn.in_transaction is False.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reflexio/server/services/storage/sqlite_storage/governance/_purge.py` around
lines 260 - 271, The purge-target persistence flow around
_record_purge_target_locked must roll back the SQLite transaction whenever the
DML operation or conn.commit() raises, then re-raise the original exception. Add
a failure-path test for this flow that verifies storage.conn.in_transaction is
False after the error.

191-230: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Acquire the SQLite write lock before each state check.

When two SQLiteStorageBase instances share a database, self._lock does not serialize them. In begin_purge_operation, the unique index prevents duplicate rows, but one concurrent caller can fail instead of receiving the existing idempotent operation. In prepare_governance_erase_targets, the later caller can overwrite advanced targets with pending and deleted_count=0.

Start BEGIN IMMEDIATE before each state check, re-check inside the transaction, and roll back before early returns. Add two-connection regression tests for both paths.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reflexio/server/services/storage/sqlite_storage/governance/_purge.py` around
lines 191 - 230, Update begin_purge_operation and
prepare_governance_erase_targets to start a BEGIN IMMEDIATE transaction before
reading existing state, so separate SQLiteStorageBase instances serialize writes
and re-check state inside the transaction. Roll back before returning an
existing idempotent operation or otherwise exiting early, and preserve advanced
erase targets instead of resetting them. Add two-connection regression tests
covering both concurrent paths.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@reflexio/server/services/playbook/service.py`:
- Around line 655-657: The synchronous finalization flows need one outer
transaction so creation and lineage updates commit atomically. In
reflexio/server/services/playbook/service.py lines 655-657, wrap
_persist_write_plan and _dispatch_playbook_schedulers in a single commit_scope
covering creation and consolidation lineage; in
reflexio/server/services/profile/service.py lines 354-355, apply the same
transaction boundary around creation and supersession. Preserve the existing
return behavior and avoid relying on the individual storage-method commits.

In `@reflexio/server/services/storage/sqlite_storage/_base.py`:
- Around line 845-859: The schema fast path in the guard around
governance_column incorrectly accepts an empty-string default for
governance_subject_ref. Reject this legacy schema when its column definition has
DEFAULT '' (or validate that existing values are nonblank) so it proceeds
through the rebuild path, and add a regression case covering the exact TEXT NOT
NULL DEFAULT '' definition.

In
`@tests/server/services/storage/test_playbook_aggregation_state_integration.py`:
- Around line 1038-1049: Update the test’s PlaybookAggregatorRequest invocation
to pass the deterministic operation_key "test-run-1", then change the
learning_meter assertion to expect request_id="test-run-1" directly instead of
reading it from learning_meter.call_args. Keep the remaining assertions
unchanged.

---

Outside diff comments:
In `@reflexio/server/services/storage/sqlite_storage/governance/_purge.py`:
- Around line 260-271: The purge-target persistence flow around
_record_purge_target_locked must roll back the SQLite transaction whenever the
DML operation or conn.commit() raises, then re-raise the original exception. Add
a failure-path test for this flow that verifies storage.conn.in_transaction is
False after the error.
- Around line 191-230: Update begin_purge_operation and
prepare_governance_erase_targets to start a BEGIN IMMEDIATE transaction before
reading existing state, so separate SQLiteStorageBase instances serialize writes
and re-check state inside the transaction. Roll back before returning an
existing idempotent operation or otherwise exiting early, and preserve advanced
erase targets instead of resetting them. Add two-connection regression tests
covering both concurrent paths.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1307db0e-796f-4f09-ba00-aeea151e8f6b

📥 Commits

Reviewing files that changed from the base of the PR and between f83d4fe and 3c054d5.

📒 Files selected for processing (23)
  • reflexio/models/api_schema/domain/entities.py
  • reflexio/server/billing_meter.py
  • reflexio/server/routes/system.py
  • reflexio/server/services/base_generation_service.py
  • reflexio/server/services/extraction/resume_worker.py
  • reflexio/server/services/playbook/components/aggregator.py
  • reflexio/server/services/playbook/service.py
  • reflexio/server/services/profile/service.py
  • reflexio/server/services/storage/sqlite_storage/_base.py
  • reflexio/server/services/storage/sqlite_storage/_governance.py
  • reflexio/server/services/storage/sqlite_storage/_session_outcomes.py
  • reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py
  • reflexio/server/services/storage/sqlite_storage/governance/_purge.py
  • reflexio/server/services/storage/storage_base/__init__.py
  • tests/server/services/extraction/test_resume_worker.py
  • tests/server/services/playbook/test_playbook_generation_service.py
  • tests/server/services/storage/sqlite_storage/test_governance_storage.py
  • tests/server/services/storage/sqlite_storage/test_session_outcome_downgrade_migration.py
  • tests/server/services/storage/test_playbook_aggregation_state_integration.py
  • tests/server/services/storage/test_storage_contract_clear_user_data.py
  • tests/server/services/storage/test_storage_contract_session_outcomes.py
  • tests/server/services/test_non_extraction_learning_metering.py
  • tests/server/test_billing_meter_events.py
🚧 Files skipped from review as they are similar to previous changes (10)
  • reflexio/models/api_schema/domain/entities.py
  • reflexio/server/services/base_generation_service.py
  • tests/server/services/test_non_extraction_learning_metering.py
  • tests/server/services/storage/test_storage_contract_session_outcomes.py
  • reflexio/server/services/extraction/resume_worker.py
  • reflexio/server/services/storage/sqlite_storage/_governance.py
  • tests/server/services/storage/sqlite_storage/test_governance_storage.py
  • reflexio/server/services/playbook/components/aggregator.py
  • reflexio/server/services/storage/sqlite_storage/_session_outcomes.py
  • reflexio/server/services/storage/sqlite_storage/governance/_erase_execution.py

Comment thread reflexio/server/services/playbook/service.py Outdated
Comment thread reflexio/server/services/storage/sqlite_storage/_base.py
Comment thread tests/server/services/storage/test_playbook_aggregation_state_integration.py Outdated
Make synchronous persistence atomic, serialize purge state transitions across SQLite connections, repair legacy outcome schemas with empty governance defaults, and strengthen metering coverage.
@yyiilluu

yyiilluu commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both outside-diff CodeRabbit findings in eebc0e5:

  • record_purge_target now rolls back on DML or commit failure, with a regression proving conn.in_transaction is false afterward.
  • begin_purge_operation and prepare_governance_erase_targets now acquire BEGIN IMMEDIATE before their state checks. Two-connection regressions prove idempotent begin retries return the existing operation and a stale prepare cannot reset an advanced target.

Validation: 301 affected-file tests passed; full non-E2E suite 5,535 passed, 73 skipped, 6 subtests; E2E 47 passed, 51 skipped; repo-wide Ruff clean; changed-file Pyright 0 errors and 0 warnings.

@yyiilluu
yyiilluu merged commit 1782ade into main Aug 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant