Skip to content

Release/5.9.4#60

Merged
aarthy-dk merged 146 commits intomainfrom
release/5.9.4
Mar 28, 2026
Merged

Release/5.9.4#60
aarthy-dk merged 146 commits intomainfrom
release/5.9.4

Conversation

@aarthy-dk
Copy link
Copy Markdown
Contributor

Features

  • ui: add flagged status to test definitions (TG-976 Phase 1) (07a5a0f)
  • ui: add notes dialog, review column, and sorting improvements (TG-980) (84b0ad1)
  • mcp: add MCP server foundation with JWT auth and ping tool (c123552)
  • mcp: add P0 tools, resources, prompts and model extensions (8f69936)
  • projects: add a project membership to handle user role in a project (6533624)
  • plugins: support multiple pages per plugin spec (6c232ef)
  • auth: scope sidebar projects to user memberships (e654258)
  • ui: add Dialog component; fix Portal stacking context (b0c96a3)
  • ui: add disabled prop to Toggle; add notIn form validator (356ba70)
  • mcp: polish inspector output and add validation (2761ffa)
  • catalog: add CSV import/export for metadata (62e490b)
  • mcp: enforce project-level permission scoping on all tools (f4e876a)
  • support oauth for databricks (cffe34c)
  • add support for Oracle 12c+ (085e45d)
  • add support for SAP HANA (4b993b5)
  • add safe_rerun to prevent data loss on Streamlit rerun (a0be715)
  • mcp: sanitize errors at tool/resource/prompt boundary (8e5e7b2)
  • pii masking, xde, hash fingerprints (8bdaa83)
  • test suites: add search filter (ffb41a9)
  • wizards: make steppers clickable (14f5e8c)

Bug Fixes

  • resolve ruff linting issues (import sorting, dict literals, unused import) (b57e70f)
  • improve upgrade commands to update revision after each script (0db6fdb)
  • ui: misc JS fixes (08a6a8a)
  • ui: mirror utils.js JSDoc type improvement to static copy (9b4e120)
  • ui: visual inconsistencies and navigation bug (e2e95df)
  • ui: hide portal when main content scrolls (3100ae7)
  • navigation: add project/permission checks and redirects (f1d29e1)
  • ui: address MR review feedback for CSV metadata import (TG-988) (cb1c9dc)
  • ui: skip entire row on CDE error, not just the field (TG-988) (e694400)
  • ui: fix auth base class kwarg, review column spacing, and grants (e3b0581)
  • ui: address MR review feedback for notes dialog (TG-976) (1f2a5f8)
  • discrepancies between flavors in hygiene issues and test types (e13a7f5)
  • sql server: make Dupe Rows test case sensitive (0e6be25)
  • discrepancies in Weekly Record Count test (911f81a)
  • edit-monitors: bugs in form - validate required fields (24e6299)
  • data catalog: remove test suite links for catalog role (351d54d)
  • update doc links (8e68494)
  • monitors: generate freshness monitors when profiling data already exists (8b39bbb)
  • use database_session() context manager for schedule query (de20ee0)
  • always clear cache in safe_rerun (671e702)
  • TG-1005: correct Daily_Record_Ct operator and Email_Format lookup regexes (adc7883)
  • TG-1005: make MSSQL calendar gap lookups consistent with other flavors (5b2d69d)
  • TG-1005: fix cross-flavor test type bugs found by validation suite (b1100e0)
  • ui: render portals/tooltips on top of Streamlit dialogs (3b18a69)
  • ui: add support for caption in select options (9e7ce46)
  • monitor: use excluded days from schedule if active (31c0361)
  • remove summary from edit table group dialog (36aa858)
  • data catalog: add help text (a326e67)
  • updates to pii masking and xdes (0146128)
  • scheduler shutdown race — check _stopping before blocking on _reload_event (4cdb3f2)
  • database urls detected as emails (592ce63)
  • data catalog: improve flag styling (39c82cf)
  • monitor generation fails to find test suite (8d3667b)
  • edge case in column history dialog (6c82a06)
  • schedules dialog: bug in pausing/deleting (7f105d5)
  • run tests: hide button in dialog after clicking link (2ead7ac)
  • table group: remove stepper from edit dialog (62e1b4c)
  • truncate timestamps to date in Daily_Record_Ct measure formula (1019526)
  • cast timestamps to date in Daily_Record_Ct source data lookup (Databricks) (25636ca)
  • ui: portals were closing when a nested portal opened (f5747aa)
  • emails: move app links to left (2512c9e)
  • update error text to be consistent (8d461c3)
  • test definitions: handle empty suite (04dbd1b)
  • scorecard: error on adding notification (b94c506)
  • copy/move tests: unique key constraints (1dfbfd9)
  • data catalog: prefix icons disappear after saving (104e0b8)
  • missing imports (66f244e)

Refactors

  • remove pydantic and streamlit-pydantic dependencies (002d5da)
  • hide MCP behind feature flag and standardize boolean settings (87f7205)
  • decouple RBAC from enterprise plugin via PluginHook (67749e4)
  • mcp: rewrite server instructions and handle plugin load errors (c73dfcc)
  • mcp: hide internal test_type codes from user-facing output (cacab09)
  • mcp: replace global admin bypass with role-based permissions (99e8781)
  • profiling: replace yaml files with TG- conditional SQL templates (f50b454)
  • flavor: make FlavorService stateless with explicit params (094a15b)
  • introduce database_session context manager (f82e34c)
  • replace st.rerun with safe_rerun in UI code (e43a3ab)
  • track writes via after_flush, clear cache in safe_rerun (292f1ab)
  • remove cache-clearing side effects from model mutations (ff7917f)
  • remove redundant cache clears from view callsites (df62103)
  • remove explicit commits from model mutations (5732304)

Miscellaneous

rboni-dk and others added 30 commits February 9, 2026 13:53
Allow users to flag test definitions for attention. The flag persists
across test runs, giving users a lightweight way to track work-in-progress
issues without needing to leave the app.

- Add `flagged` boolean column to test_definitions table
- Add flag/unflag disposition actions on Test Definitions and Test Results pages
- Add "Flagged" filter on both pages
- Add flagged toggle to test definition edit form
- Include flagged column in grid display, detail panel, and Excel export
- Fix pre-existing bug where inherited dataclass fields were dropped by to_dataframe

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…TG-980)

- Notes dialog: styled subtitle (Table/Column/Test), tooltip fix, auto-expand when empty, form reset on add/edit
- Test results: merged Review column (disposition + flagged + notes count), Flagged/Has Notes sorting
- Test definitions: notes count column, Flagged/Has Notes sorting
- Model: add get_notes_count_by_ids, use uuid4 for TestDefinitionNote id
- Grid: add column_styles parameter to render_grid_select

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ests

Extract pure decision/transformation logic from I/O-heavy command functions
into standalone functions, then add unit tests for each. No behavioral changes.

Extractions:
- calculate_sampling_params() from run_profiling.py into profiling_query.py
- collect_test_identifiers() and check_identifiers() from run_test_validation.py
- build_cat_expressions(), group_cat_tests(), parse_cat_results() from execute_tests_query.py
- calculate_prediction_tolerances() and Z_SCORE_MAP from test_thresholds_prediction.py

Tests for _score_card_to_results() added directly (already a standalone function).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ed import)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both pydantic (v1) and streamlit-pydantic are unused — the only consumer
(ui/forms.py) has zero imports across the codebase. Removing clears the
path for PR 1 to introduce pydantic v2 as a transitive dep of the MCP SDK.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add FastMCP server with streamable HTTP transport, JWT authentication
reusing existing token schema, and a smoke-test ping tool. Extract
shared auth building blocks (JWT signing, password verification,
permission checking) from UI into common/auth.py so both the Streamlit
UI and MCP server use the same logic.

- Add mcp[cli], uvicorn, PyJWT, bcrypt to core dependencies
- Add MCP_PORT, MCP_HOST, MCP_ENABLED settings
- Create common/auth.py with shared JWT and password utilities
- Refactor ui/auth.py to use common/auth.py
- Create mcp/server.py with FastMCP app and ping tool
- Create mcp/auth.py for MCP-specific authenticate/validate flows
- Add `testgen run-app mcp` and `testgen mcp-token` CLI commands
- Add 17 unit tests (9 common auth + 8 MCP auth)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Default MCP_ENABLED to "no" until the feature is complete. Standardize
all boolean env var checks to accept ("yes", "true") only — drop "y".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sync main to enterprise

See merge request dkinternal/testgen/dataops-testgen!422
Remove direct import of testgen_enterprise_auth from OS auth code.
RBAC is now discovered through the plugin system: plugins declare an
RBACProvider subclass on their PluginSpec, and PluginHook resolves it
at startup. OS default grants all permissions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- 8 MCP tools: discovery (inventory, projects, suites), test runs
  (recent runs), test results (results, failure summary, history),
  reference (test type lookup)
- 2 resources: test-types reference table, glossary
- 4 prompts: health_check, investigate_failures, table_health, compare_runs
- Model extensions: TestResult (select_results, select_failures,
  select_history), DataTable (new partial model for data_table_chars)
- Inventory service with ORM queries and adaptive compact mode
- 53 unit tests for all new modules

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Simplify MCP instructions to focus on data model and navigation.
Gracefully skip plugins that fail to load in MCP context (e.g. when
Streamlit v2 component registration is unavailable).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(mcp): add MCP server foundation with JWT auth and ping tool

See merge request dkinternal/testgen/dataops-testgen!414
CI and upgrade improvements

See merge request dkinternal/testgen/dataops-testgen!423
PluginSpec.page (single class) replaced with PluginSpec.pages (list),
allowing a plugin to register more than one Page in a single spec.
Application.global_admin_paths now collects paths whose permission is
'global_admin' so the sidebar can switch into the admin context.
- Filter the project dropdown to only projects the user is a member of
- Call load_user_role on every render so role reflects current project
- Introduce 'global_admin' permission; sidebar enters admin-only context
  on global_admin pages (no project nav, AdminMenuItem items, AdminCTA)
- is_global_admin prop lets non-admin users see the admin CTA in project context
Portal now appends to document.body so position:absolute is
document-relative, preventing clip by positioned ancestors.  Outside-
click and mutual-exclusion logic moved into the component itself.

Dialog wraps Portal to provide a modal overlay with a title bar and
close button, used by the project settings members form.
Toggle accepts a disabled prop that greys out the input and suppresses
the clickable cursor, consistent with other form components.

notIn validator rejects values found in a provided list, with an
optional formatter and custom error message — used for unique-name
validation on the project settings form.
- Table: use .val when reading dataColumns.length for colspan so it
  reacts to state updates rather than capturing the initial value
- Streamlit: add disableV2() to clean up v2 state on component teardown
- data_catalog: import DISABLED_ACTION_TEXT from display_utils
- utils.js: improve JSDoc type for getValue()
aarthy-dk and others added 26 commits March 26, 2026 00:31
Misc QA fixes + doc updates

See merge request dkinternal/testgen/dataops-testgen!446
COUNT(DISTINCT col) on timestamp columns counts unique timestamps, not
unique dates, producing negative measures (e.g., -1575 instead of 240).
Add CAST(col AS DATE) for 7 flavors that were missing date truncation:
postgresql, snowflake, databricks, redshift, redshift_spectrum, mssql, trino.
BigQuery, Oracle, and SAP HANA already truncated correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…atabricks)

date_bounds CTE used raw MIN/MAX without CAST, so SEQUENCE generated
timestamps that never matched the date-typed existing_periods — all
dates appeared missing for timestamp columns.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…erprise'

fix(TG-1005): Daily_Record_Ct date truncation + CI fix + timestamp test coverage

See merge request dkinternal/testgen/dataops-testgen!443
fix(ui): portals were closing when a nested portal opened

See merge request dkinternal/testgen/dataops-testgen!447
QA fixes

See merge request dkinternal/testgen/dataops-testgen!448
QA fixes + docs changes

See merge request dkinternal/testgen/dataops-testgen!449
@aarthy-dk aarthy-dk merged commit 741ec5b into main Mar 28, 2026
2 checks passed
@aarthy-dk aarthy-dk deleted the release/5.9.4 branch March 28, 2026 03:37
@github-actions
Copy link
Copy Markdown

Coverage

Tests Skipped Failures Errors Time
5 0 💤 0 ❌ 5 🔥 13.022s ⏱️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants